2024-04-24T23:27:11.004 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T23:27:11.005 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204 --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} --verbose -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/orig.config.yaml 2024-04-24T23:27:11.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:27:11.159 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672204' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfheErB+R/oCdncN1/pPykxwLFnjM5IM30LCb/S0LxDSevynN+BqvxCS+6ndFqAQU3v1z412hIl6vdYAqUHhN0= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAF70IA5khwXxJlW8T9qqI7TDcfoN9iBzK/+MXAbEU4HuON8jhpi4/YMAW0JJEHEGmVrYTvtnV2p52yIUUrxQhc= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIghVqNt56MLKC5LeSYAsqRNs2SLIjnxurC93VVw4zucgIm1FMSBIC7J5M6sMmsmZ7/+irwPzyA/L48CgBvHPt0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T23:27:11.159 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-24T23:27:11.160 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-24T23:27:11.160 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T23:27:11.161 INFO:teuthology.task.internal:Checking packages... 2024-04-24T23:27:11.186 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-24T23:27:11.186 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T23:27:11.186 INFO:teuthology.packaging:ref: None 2024-04-24T23:27:11.186 INFO:teuthology.packaging:tag: None 2024-04-24T23:27:11.186 INFO:teuthology.packaging:branch: squid 2024-04-24T23:27:11.186 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:27:11.186 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-24T23:27:11.402 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-24T23:27:11.404 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T23:27:11.410 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T23:27:11.410 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T23:27:11.415 INFO:teuthology.task.internal:Saving configuration 2024-04-24T23:27:11.435 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T23:27:11.441 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T23:27:11.467 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:22:10.852512', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfheErB+R/oCdncN1/pPykxwLFnjM5IM30LCb/S0LxDSevynN+BqvxCS+6ndFqAQU3v1z412hIl6vdYAqUHhN0='} 2024-04-24T23:27:11.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:22:10.850524', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAF70IA5khwXxJlW8T9qqI7TDcfoN9iBzK/+MXAbEU4HuON8jhpi4/YMAW0JJEHEGmVrYTvtnV2p52yIUUrxQhc='} 2024-04-24T23:27:11.514 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:22:10.854237', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIghVqNt56MLKC5LeSYAsqRNs2SLIjnxurC93VVw4zucgIm1FMSBIC7J5M6sMmsmZ7/+irwPzyA/L48CgBvHPt0='} 2024-04-24T23:27:11.514 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T23:27:11.523 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-24T23:27:11.524 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-24T23:27:11.524 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-24T23:27:11.524 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T23:27:11.627 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f560ed2c160>, signals=[15]) 2024-04-24T23:27:11.628 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T23:27:11.634 INFO:teuthology.task.internal:Opening connections... 2024-04-24T23:27:11.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:27:11.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:27:11.714 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-04-24T23:27:11.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:27:11.789 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-04-24T23:27:11.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:27:11.865 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T23:27:11.881 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-24T23:27:11.899 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-24T23:27:11.899 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-24T23:27:11.957 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-24T23:27:11.957 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-24T23:27:11.957 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-24T23:27:11.957 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:27:11.957 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:27:11.958 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:27:11.959 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-24T23:27:11.986 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-04-24T23:27:12.004 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-04-24T23:27:12.004 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-04-24T23:27:12.061 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-04-24T23:27:12.061 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-04-24T23:27:12.061 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-04-24T23:27:12.061 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:27:12.062 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:27:12.062 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-04-24T23:27:12.085 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-04-24T23:27:12.103 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-04-24T23:27:12.104 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:27:12.160 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:27:12.161 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:27:12.161 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-04-24T23:27:12.185 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T23:27:12.197 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T23:27:12.202 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T23:27:12.203 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:27:12.205 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:27:12.208 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:27:12.223 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T23:27:12.228 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T23:27:12.229 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:27:12.262 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:27:12.264 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:27:12.357 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T23:27:12.431 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T23:27:12.447 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-24T23:27:12.447 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-24T23:27:12.447 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-24T23:27:12.447 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:27:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:12.464 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-24T23:27:12.522 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-437.el9.x86_64 2024-04-24T23:27:12.522 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-437.el9.x86_64 2024-04-24T23:27:12.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-04-24T23:27:14.726 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T23:27:17.494 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-24T23:27:21.953 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-04-24T23:27:23.862 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-24T23:27:25.479 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-24T23:27:30.570 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-24T23:27:32.427 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T23:27:32.427 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T23:27:32.514 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-24T23:27:32.515 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-24T23:27:32.515 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-24T23:27:32.622 DEBUG:teuthology.orchestra.run.smithi107:> echo no | sudo yum reinstall kernel || true 2024-04-24T23:27:33.116 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:27:30 PM UTC. 2024-04-24T23:27:33.234 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:27:33.242 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 255 k 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2024-04-24T23:27:33.243 INFO:teuthology.orchestra.run.smithi107.stderr:Operation aborted. 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi107.stdout:Is this ok [y/N]: 2024-04-24T23:27:33.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum reinstall -y kernel || true 2024-04-24T23:27:33.806 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:27:30 PM UTC. 2024-04-24T23:27:33.922 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 255 k 2024-04-24T23:27:33.923 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2024-04-24T23:27:33.924 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-24T23:27:34.747 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-437.el9.x86_64.rpm 359 kB/s | 255 kB 00:00 2024-04-24T23:27:34.747 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:27:34.748 INFO:teuthology.orchestra.run.smithi107.stdout:Total 309 kB/s | 255 kB 00:00 2024-04-24T23:27:34.767 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-24T23:27:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-24T23:27:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-24T23:27:34.946 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-24T23:27:34.946 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-24T23:27:35.077 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-24T23:27:35.156 INFO:teuthology.orchestra.run.smithi107.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:27:35.157 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:27:35.800 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:27:35.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:27:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:27:36.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:36.511 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalled: 2024-04-24T23:27:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T23:27:36.511 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:27:36.511 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-24T23:27:36.701 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T23:27:36.746 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T23:27:36.747 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T23:27:36.747 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T23:27:36.747 DEBUG:teuthology.parallel:result is None 2024-04-24T23:27:36.747 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T23:27:36.756 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T23:27:36.756 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:27:36.759 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:27:36.761 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:27:36.779 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T23:27:36.786 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T23:27:36.792 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T23:27:36.792 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:27:36.813 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:27:36.818 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:27:36.882 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T23:27:36.888 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T23:27:36.889 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:27:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:36.905 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:27:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:36.922 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:27:36.924 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:27:36.948 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:27:36.955 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:36.970 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:36.981 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:36.998 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:37.013 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:37.028 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:27:37.030 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T23:27:37.037 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T23:27:37.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:27:37.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:27:37.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:27:37.070 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T23:27:37.079 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T23:27:37.079 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:27:37.104 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:27:37.109 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:27:37.126 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.239 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.358 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:27:37.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:27:37.421 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.506 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.628 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:27:37.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:27:37.691 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:27:37.780 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:27:37.906 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:27:37.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:27:37.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-04-24T23:27:37.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-04-24T23:27:37.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-04-24T23:27:38.012 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:27:38.016 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:27:38.063 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:27:38.419 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T23:27:38.426 INFO:teuthology.task.internal:Starting timer... 2024-04-24T23:27:38.426 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T23:27:38.446 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T23:27:38.455 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-24T23:27:38.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-04-24T23:27:38.510 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-04-24T23:27:38.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-04-24T23:27:38.569 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-04-24T23:27:38.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-04-24T23:27:38.647 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-04-24T23:27:38.649 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T23:27:38.649 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-04-24T23:27:38.673 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-04-24T23:27:38.673 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-04-24T23:27:38.725 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-04-24T23:27:38.725 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-04-24T23:27:38.756 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-04-24T23:27:38.756 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-04-24T23:27:38.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:27:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:38.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:27:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:38.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:27:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:27:38.832 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T23:27:38.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:27:38.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:27:38.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:27:38.919 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T23:27:38.928 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T23:27:38.934 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T23:27:39.061 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T23:27:39.077 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T23:27:39.078 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi170.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T23:31:52.274 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-04-24T23:31:52.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-24T23:31:52.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:31:52.354 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-24T23:31:52.437 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-24T23:31:52.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-04-24T23:31:52.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:31:52.513 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-04-24T23:31:52.555 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-04-24T23:31:52.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-24T23:31:52.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:31:52.629 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-04-24T23:31:52.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-04-24T23:31:52.719 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T23:31:52.731 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T23:31:52.732 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:31:52.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:31:52.735 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:31:52.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:31:52.737 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:31:52.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:31:52.761 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:31:52.764 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:31:52.774 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:31:52.778 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:31:52.787 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:31:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-04-24T23:31:52.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-04-24T23:31:52.800 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:31:52.806 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-04-24T23:31:52.810 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-04-24T23:31:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:31:52.824 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:31:52.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-04-24T23:31:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:31:52.834 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-04-24T23:31:52.836 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:31:52.860 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:31:52.888 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:31:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:31:52.891 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:31:52.891 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-04-24T23:31:52.891 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.891 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.892 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.892 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.923 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.927 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:31:52.928 INFO:teuthology.run_tasks:Running task install... 2024-04-24T23:31:52.937 DEBUG:teuthology.task.install:project ceph 2024-04-24T23:31:52.938 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-24T23:31:52.938 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-24T23:31:52.938 INFO:teuthology.task.install:Using flavor: default 2024-04-24T23:31:52.946 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T23:31:52.946 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T23:31:52.946 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:31:52.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:31:52.949 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:31:52.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:31:52.950 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:31:52.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:31:53.102 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:31:53.102 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:31:53.172 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:31:53.173 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:31:53.178 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:31:53.178 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:31:53.274 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:31:53.274 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:31:53.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:31:53.285 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:31:53.286 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:31:53.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:31:53.299 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:31:53.299 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:31:53.311 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:31:53.311 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:31:53.340 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:31:53.340 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:31:53.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:31:53.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:31:53.365 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:31:53.365 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:31:53.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:31:53.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:31:53.437 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:31:53.449 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:31:53.503 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:31:53.504 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-04-24T23:31:53.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-04-24T23:31:53.512 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-04-24T23:31:53.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-04-24T23:31:53.566 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-04-24T23:31:53.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-04-24T23:31:53.726 INFO:teuthology.orchestra.run.smithi195.stdout:48 files removed 2024-04-24T23:31:53.738 INFO:teuthology.orchestra.run.smithi170.stdout:48 files removed 2024-04-24T23:31:53.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:31:53.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:31:53.791 INFO:teuthology.orchestra.run.smithi107.stdout:48 files removed 2024-04-24T23:31:53.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:31:54.410 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-04-24T23:31:54.425 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 238 kB/s | 83 kB 00:00 2024-04-24T23:31:54.480 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 185 kB/s | 83 kB 00:00 2024-04-24T23:31:54.752 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-04-24T23:31:54.784 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-24T23:31:54.914 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-24T23:31:55.134 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-24T23:31:55.162 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-24T23:31:55.281 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-04-24T23:31:55.685 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-24T23:31:56.898 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-04-24T23:31:57.011 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-04-24T23:31:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 19 MB 00:01 2024-04-24T23:31:59.096 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-24T23:32:00.307 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-04-24T23:32:03.277 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-04-24T23:32:03.897 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.2 MB 00:01 2024-04-24T23:32:05.028 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-04-24T23:32:05.414 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-04-24T23:32:06.624 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-04-24T23:32:07.101 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-24T23:32:08.098 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-24T23:32:08.924 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-24T23:32:10.862 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-24T23:32:12.257 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-24T23:32:13.229 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-04-24T23:32:14.376 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:14.376 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:14.387 INFO:teuthology.orchestra.run.smithi195.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:32:14.387 INFO:teuthology.orchestra.run.smithi195.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:32:14.388 INFO:teuthology.orchestra.run.smithi195.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:32:14.388 INFO:teuthology.orchestra.run.smithi195.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:32:14.390 INFO:teuthology.orchestra.run.smithi195.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:32:14.393 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:32:14.393 INFO:teuthology.orchestra.run.smithi195.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:32:14.452 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-04-24T23:32:14.467 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:32:14.468 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:32:14.469 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:14.470 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:32:14.471 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:32:14.472 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:32:14.473 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:32:14.474 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:32:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:32:14.476 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:32:14.477 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:32:14.478 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout:Install 148 Packages 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 4 Packages 2024-04-24T23:32:14.479 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:32:14.480 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 220 M 2024-04-24T23:32:14.480 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-04-24T23:32:15.372 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:15.373 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:15.385 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:32:15.385 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:32:15.386 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:32:15.387 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:32:15.388 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:32:15.391 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:32:15.391 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:32:15.453 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-24T23:32:15.462 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-24T23:32:15.462 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-24T23:32:15.462 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-24T23:32:15.462 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-24T23:32:15.462 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:32:15.463 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:32:15.464 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:32:15.465 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:32:15.466 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:32:15.466 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:32:15.466 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:32:15.466 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:32:15.466 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:32:15.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:32:15.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:32:15.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:32:15.467 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:32:15.467 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:32:15.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:32:15.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:32:15.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:32:15.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:32:15.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:32:15.470 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:32:15.471 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:32:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:32:15.473 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:32:15.474 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:32:15.474 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:32:15.474 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:32:15.474 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:32:15.474 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:32:15.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:32:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:32:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:32:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:32:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:32:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:32:15.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:32:15.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:32:15.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:32:15.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:32:15.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:32:15.479 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:32:15.479 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:32:15.481 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:32:15.481 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:32:15.481 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:32:15.481 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:32:15.482 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:32:15.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:32:15.484 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:Install 148 Packages 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 4 Packages 2024-04-24T23:32:15.485 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:32:15.486 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 220 M 2024-04-24T23:32:15.486 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-24T23:32:15.875 INFO:teuthology.orchestra.run.smithi195.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 12 kB/s | 6.6 kB 00:00 2024-04-24T23:32:16.009 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-24T23:32:16.242 INFO:teuthology.orchestra.run.smithi195.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 5.7 MB/s | 5.4 MB 00:00 2024-04-24T23:32:16.301 INFO:teuthology.orchestra.run.smithi195.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.4 MB/s | 145 kB 00:00 2024-04-24T23:32:16.471 INFO:teuthology.orchestra.run.smithi107.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-24T23:32:16.576 INFO:teuthology.orchestra.run.smithi195.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-04-24T23:32:16.793 INFO:teuthology.orchestra.run.smithi195.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-24T23:32:16.860 INFO:teuthology.orchestra.run.smithi195.stdout:(6/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 904 kB/s | 890 kB 00:00 2024-04-24T23:32:17.063 INFO:teuthology.orchestra.run.smithi107.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.5 MB/s | 890 kB 00:00 2024-04-24T23:32:17.155 INFO:teuthology.orchestra.run.smithi107.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.5 MB/s | 145 kB 00:00 2024-04-24T23:32:17.385 INFO:teuthology.orchestra.run.smithi195.stdout:(7/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 10 MB/s | 21 MB 00:02 2024-04-24T23:32:17.511 INFO:teuthology.orchestra.run.smithi195.stdout:(8/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 6.5 MB/s | 4.7 MB 00:00 2024-04-24T23:32:17.555 INFO:teuthology.orchestra.run.smithi107.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 5.9 MB/s | 2.4 MB 00:00 2024-04-24T23:32:17.577 INFO:teuthology.orchestra.run.smithi195.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 385 kB/s | 25 kB 00:00 2024-04-24T23:32:18.014 INFO:teuthology.orchestra.run.smithi107.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-04-24T23:32:18.118 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:18.119 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:32:18.130 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:32:18.130 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:32:18.131 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:32:18.132 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:32:18.133 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:32:18.136 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:32:18.136 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:32:18.196 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-04-24T23:32:18.210 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:32:18.211 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:32:18.212 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:18.213 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:32:18.214 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:32:18.215 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:32:18.216 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:32:18.217 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:32:18.218 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:32:18.219 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:32:18.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:32:18.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:32:18.222 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-04-24T23:32:18.223 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-04-24T23:32:18.223 INFO:teuthology.orchestra.run.smithi170.stdout:Install 148 Packages 2024-04-24T23:32:18.223 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 4 Packages 2024-04-24T23:32:18.223 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:32:18.223 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 220 M 2024-04-24T23:32:18.224 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-04-24T23:32:18.311 INFO:teuthology.orchestra.run.smithi195.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 11 MB/s | 10 MB 00:00 2024-04-24T23:32:18.453 INFO:teuthology.orchestra.run.smithi195.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.5 MB/s | 222 kB 00:00 2024-04-24T23:32:18.528 INFO:teuthology.orchestra.run.smithi195.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 362 kB/s | 27 kB 00:00 2024-04-24T23:32:18.603 INFO:teuthology.orchestra.run.smithi195.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 424 kB/s | 32 kB 00:00 2024-04-24T23:32:18.705 INFO:teuthology.orchestra.run.smithi107.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 6.8 MB/s | 4.7 MB 00:00 2024-04-24T23:32:18.887 INFO:teuthology.orchestra.run.smithi195.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 2.6 MB/s | 745 kB 00:00 2024-04-24T23:32:18.979 INFO:teuthology.orchestra.run.smithi195.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.7 MB/s | 163 kB 00:00 2024-04-24T23:32:19.071 INFO:teuthology.orchestra.run.smithi195.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.3 MB/s | 127 kB 00:00 2024-04-24T23:32:19.313 INFO:teuthology.orchestra.run.smithi195.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 2.0 MB/s | 501 kB 00:00 2024-04-24T23:32:19.494 INFO:teuthology.orchestra.run.smithi170.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-24T23:32:20.370 INFO:teuthology.orchestra.run.smithi170.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.0 MB/s | 890 kB 00:00 2024-04-24T23:32:20.461 INFO:teuthology.orchestra.run.smithi170.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.5 MB/s | 145 kB 00:00 2024-04-24T23:32:20.564 INFO:teuthology.orchestra.run.smithi107.stdout:(7/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 9.0 MB/s | 17 MB 00:01 2024-04-24T23:32:20.647 INFO:teuthology.orchestra.run.smithi107.stdout:(8/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-04-24T23:32:20.760 INFO:teuthology.orchestra.run.smithi107.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 232 kB/s | 25 kB 00:00 2024-04-24T23:32:21.380 INFO:teuthology.orchestra.run.smithi195.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-04-24T23:32:21.411 INFO:teuthology.orchestra.run.smithi170.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-04-24T23:32:21.472 INFO:teuthology.orchestra.run.smithi195.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 493 kB/s | 45 kB 00:00 2024-04-24T23:32:21.563 INFO:teuthology.orchestra.run.smithi195.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-24T23:32:21.664 INFO:teuthology.orchestra.run.smithi195.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.6 MB/s | 162 kB 00:00 2024-04-24T23:32:21.836 INFO:teuthology.orchestra.run.smithi170.stdout:(5/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-24T23:32:21.842 INFO:teuthology.orchestra.run.smithi195.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.8 MB/s | 321 kB 00:00 2024-04-24T23:32:21.912 INFO:teuthology.orchestra.run.smithi170.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-04-24T23:32:21.939 INFO:teuthology.orchestra.run.smithi107.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 7.4 MB/s | 10 MB 00:01 2024-04-24T23:32:21.940 INFO:teuthology.orchestra.run.smithi195.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.9 MB/s | 299 kB 00:00 2024-04-24T23:32:22.015 INFO:teuthology.orchestra.run.smithi195.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-24T23:32:22.022 INFO:teuthology.orchestra.run.smithi107.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.6 MB/s | 222 kB 00:00 2024-04-24T23:32:22.082 INFO:teuthology.orchestra.run.smithi195.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-24T23:32:22.098 INFO:teuthology.orchestra.run.smithi107.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 360 kB/s | 27 kB 00:00 2024-04-24T23:32:22.173 INFO:teuthology.orchestra.run.smithi107.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 423 kB/s | 32 kB 00:00 2024-04-24T23:32:22.349 INFO:teuthology.orchestra.run.smithi107.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.1 MB/s | 745 kB 00:00 2024-04-24T23:32:22.441 INFO:teuthology.orchestra.run.smithi107.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.7 MB/s | 163 kB 00:00 2024-04-24T23:32:22.533 INFO:teuthology.orchestra.run.smithi107.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.3 MB/s | 127 kB 00:00 2024-04-24T23:32:23.050 INFO:teuthology.orchestra.run.smithi107.stdout:(17/152): ceph-common-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 21 MB 00:06 2024-04-24T23:32:23.109 INFO:teuthology.orchestra.run.smithi107.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 871 kB/s | 501 kB 00:00 2024-04-24T23:32:23.171 INFO:teuthology.orchestra.run.smithi107.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 775 kB/s | 45 kB 00:00 2024-04-24T23:32:23.171 INFO:teuthology.orchestra.run.smithi170.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 3.5 MB/s | 4.7 MB 00:01 2024-04-24T23:32:23.182 INFO:teuthology.orchestra.run.smithi195.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 2.8 MB/s | 3.1 MB 00:01 2024-04-24T23:32:23.217 INFO:teuthology.orchestra.run.smithi107.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 2.6 MB/s | 133 kB 00:00 2024-04-24T23:32:23.284 INFO:teuthology.orchestra.run.smithi107.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-24T23:32:23.307 INFO:teuthology.orchestra.run.smithi195.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.3 MB/s | 171 kB 00:00 2024-04-24T23:32:23.360 INFO:teuthology.orchestra.run.smithi107.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-24T23:32:23.374 INFO:teuthology.orchestra.run.smithi195.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 476 kB/s | 32 kB 00:00 2024-04-24T23:32:23.443 INFO:teuthology.orchestra.run.smithi107.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-24T23:32:23.466 INFO:teuthology.orchestra.run.smithi195.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.5 MB/s | 141 kB 00:00 2024-04-24T23:32:23.527 INFO:teuthology.orchestra.run.smithi107.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-24T23:32:23.610 INFO:teuthology.orchestra.run.smithi107.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-24T23:32:24.244 INFO:teuthology.orchestra.run.smithi107.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 4.9 MB/s | 3.1 MB 00:00 2024-04-24T23:32:24.336 INFO:teuthology.orchestra.run.smithi107.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-24T23:32:24.420 INFO:teuthology.orchestra.run.smithi107.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 381 kB/s | 32 kB 00:00 2024-04-24T23:32:24.503 INFO:teuthology.orchestra.run.smithi107.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.7 MB/s | 141 kB 00:00 2024-04-24T23:32:24.629 INFO:teuthology.orchestra.run.smithi107.stdout:(30/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 3.4 MB/s | 5.3 MB 00:01 2024-04-24T23:32:25.008 INFO:teuthology.orchestra.run.smithi195.stdout:(30/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 2.3 MB/s | 3.5 MB 00:01 2024-04-24T23:32:25.229 INFO:teuthology.orchestra.run.smithi107.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.8 MB/s | 3.5 MB 00:00 2024-04-24T23:32:25.279 INFO:teuthology.orchestra.run.smithi107.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 4.9 MB/s | 249 kB 00:00 2024-04-24T23:32:25.338 INFO:teuthology.orchestra.run.smithi107.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 846 kB/s | 49 kB 00:00 2024-04-24T23:32:25.396 INFO:teuthology.orchestra.run.smithi107.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 263 kB/s | 15 kB 00:00 2024-04-24T23:32:25.455 INFO:teuthology.orchestra.run.smithi107.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 4.4 MB/s | 264 kB 00:00 2024-04-24T23:32:25.479 INFO:teuthology.orchestra.run.smithi170.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 4.7 MB/s | 17 MB 00:03 2024-04-24T23:32:25.572 INFO:teuthology.orchestra.run.smithi107.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-24T23:32:25.591 INFO:teuthology.orchestra.run.smithi195.stdout:(31/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 6.0 MB/s | 48 MB 00:08 2024-04-24T23:32:25.622 INFO:teuthology.orchestra.run.smithi107.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 367 kB/s | 18 kB 00:00 2024-04-24T23:32:25.658 INFO:teuthology.orchestra.run.smithi195.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.7 MB/s | 249 kB 00:00 2024-04-24T23:32:25.687 INFO:teuthology.orchestra.run.smithi170.stdout:(9/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 3.1 MB/s | 21 MB 00:06 2024-04-24T23:32:25.701 INFO:teuthology.orchestra.run.smithi195.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 1.2 MB/s | 49 kB 00:00 2024-04-24T23:32:25.704 INFO:teuthology.orchestra.run.smithi170.stdout:(10/152): ceph-selinux-19.0.0-2445.gd2c45d97.el 112 kB/s | 25 kB 00:00 2024-04-24T23:32:25.730 INFO:teuthology.orchestra.run.smithi107.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-04-24T23:32:25.742 INFO:teuthology.orchestra.run.smithi195.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 370 kB/s | 15 kB 00:00 2024-04-24T23:32:25.821 INFO:teuthology.orchestra.run.smithi170.stdout:(11/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 3.9 MB/s | 10 MB 00:02 2024-04-24T23:32:25.826 INFO:teuthology.orchestra.run.smithi195.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-24T23:32:25.839 INFO:teuthology.orchestra.run.smithi107.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-04-24T23:32:25.854 INFO:teuthology.orchestra.run.smithi170.stdout:(12/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.4 MB/s | 222 kB 00:00 2024-04-24T23:32:25.871 INFO:teuthology.orchestra.run.smithi170.stdout:(13/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 541 kB/s | 27 kB 00:00 2024-04-24T23:32:25.913 INFO:teuthology.orchestra.run.smithi170.stdout:(14/152): libcephfs-devel-19.0.0-2445.gd2c45d97 541 kB/s | 32 kB 00:00 2024-04-24T23:32:25.931 INFO:teuthology.orchestra.run.smithi107.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 5.7 MB/s | 7.4 MB 00:01 2024-04-24T23:32:25.951 INFO:teuthology.orchestra.run.smithi195.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.0 MB/s | 762 kB 00:00 2024-04-24T23:32:25.956 INFO:teuthology.orchestra.run.smithi107.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-24T23:32:25.980 INFO:teuthology.orchestra.run.smithi170.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 163 kB 00:00 2024-04-24T23:32:26.001 INFO:teuthology.orchestra.run.smithi195.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 366 kB/s | 18 kB 00:00 2024-04-24T23:32:26.023 INFO:teuthology.orchestra.run.smithi107.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-24T23:32:26.031 INFO:teuthology.orchestra.run.smithi170.stdout:(16/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.6 MB/s | 745 kB 00:00 2024-04-24T23:32:26.057 INFO:teuthology.orchestra.run.smithi170.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 1.6 MB/s | 127 kB 00:00 2024-04-24T23:32:26.065 INFO:teuthology.orchestra.run.smithi107.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-24T23:32:26.090 INFO:teuthology.orchestra.run.smithi107.stdout:(44/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-04-24T23:32:26.110 INFO:teuthology.orchestra.run.smithi195.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 742 kB/s | 80 kB 00:00 2024-04-24T23:32:26.116 INFO:teuthology.orchestra.run.smithi107.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-24T23:32:26.135 INFO:teuthology.orchestra.run.smithi107.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-24T23:32:26.157 INFO:teuthology.orchestra.run.smithi170.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 3.9 MB/s | 501 kB 00:00 2024-04-24T23:32:26.158 INFO:teuthology.orchestra.run.smithi107.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-04-24T23:32:26.218 INFO:teuthology.orchestra.run.smithi195.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2024-04-24T23:32:26.232 INFO:teuthology.orchestra.run.smithi170.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 602 kB/s | 45 kB 00:00 2024-04-24T23:32:26.260 INFO:teuthology.orchestra.run.smithi195.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-24T23:32:26.302 INFO:teuthology.orchestra.run.smithi195.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-04-24T23:32:26.307 INFO:teuthology.orchestra.run.smithi170.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.7 MB/s | 133 kB 00:00 2024-04-24T23:32:26.369 INFO:teuthology.orchestra.run.smithi195.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-24T23:32:26.383 INFO:teuthology.orchestra.run.smithi170.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.1 MB/s | 162 kB 00:00 2024-04-24T23:32:26.430 INFO:teuthology.orchestra.run.smithi195.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 106 kB 00:00 2024-04-24T23:32:26.466 INFO:teuthology.orchestra.run.smithi170.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.8 MB/s | 321 kB 00:00 2024-04-24T23:32:26.470 INFO:teuthology.orchestra.run.smithi195.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.3 MB/s | 135 kB 00:00 2024-04-24T23:32:26.512 INFO:teuthology.orchestra.run.smithi195.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-24T23:32:26.550 INFO:teuthology.orchestra.run.smithi170.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-24T23:32:26.554 INFO:teuthology.orchestra.run.smithi195.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-04-24T23:32:26.596 INFO:teuthology.orchestra.run.smithi195.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 16 MB/s | 689 kB 00:00 2024-04-24T23:32:26.609 INFO:teuthology.orchestra.run.smithi170.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.7 MB/s | 99 kB 00:00 2024-04-24T23:32:26.646 INFO:teuthology.orchestra.run.smithi195.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 19 MB/s | 943 kB 00:00 2024-04-24T23:32:26.675 INFO:teuthology.orchestra.run.smithi170.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-24T23:32:26.688 INFO:teuthology.orchestra.run.smithi195.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-24T23:32:26.739 INFO:teuthology.orchestra.run.smithi195.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-24T23:32:26.823 INFO:teuthology.orchestra.run.smithi195.stdout:(51/152): ceph-mgr-diskprediction-local-19.0.0- 4.1 MB/s | 7.4 MB 00:01 2024-04-24T23:32:26.834 INFO:teuthology.orchestra.run.smithi170.stdout:(26/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 6.8 MB/s | 5.3 MB 00:00 2024-04-24T23:32:26.892 INFO:teuthology.orchestra.run.smithi107.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 908 kB/s | 689 kB 00:00 2024-04-24T23:32:26.901 INFO:teuthology.orchestra.run.smithi170.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-24T23:32:26.960 INFO:teuthology.orchestra.run.smithi170.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 545 kB/s | 32 kB 00:00 2024-04-24T23:32:26.981 INFO:teuthology.orchestra.run.smithi195.stdout:(52/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 1.7 MB/s | 17 MB 00:10 2024-04-24T23:32:27.006 INFO:teuthology.orchestra.run.smithi195.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-24T23:32:27.008 INFO:teuthology.orchestra.run.smithi107.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 917 kB/s | 107 kB 00:00 2024-04-24T23:32:27.027 INFO:teuthology.orchestra.run.smithi170.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.1 MB/s | 141 kB 00:00 2024-04-24T23:32:27.034 INFO:teuthology.orchestra.run.smithi107.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-24T23:32:27.040 INFO:teuthology.orchestra.run.smithi195.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 151 kB/s | 33 kB 00:00 2024-04-24T23:32:27.065 INFO:teuthology.orchestra.run.smithi195.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-24T23:32:27.091 INFO:teuthology.orchestra.run.smithi195.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-04-24T23:32:27.109 INFO:teuthology.orchestra.run.smithi107.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-04-24T23:32:27.116 INFO:teuthology.orchestra.run.smithi195.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-24T23:32:27.143 INFO:teuthology.orchestra.run.smithi107.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-24T23:32:27.149 INFO:teuthology.orchestra.run.smithi195.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-04-24T23:32:27.183 INFO:teuthology.orchestra.run.smithi195.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-24T23:32:27.194 INFO:teuthology.orchestra.run.smithi170.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 6.0 MB/s | 3.1 MB 00:00 2024-04-24T23:32:27.201 INFO:teuthology.orchestra.run.smithi107.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-04-24T23:32:27.225 INFO:teuthology.orchestra.run.smithi195.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 613 kB/s | 25 kB 00:00 2024-04-24T23:32:27.250 INFO:teuthology.orchestra.run.smithi195.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-24T23:32:27.275 INFO:teuthology.orchestra.run.smithi195.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-24T23:32:27.301 INFO:teuthology.orchestra.run.smithi195.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-24T23:32:27.302 INFO:teuthology.orchestra.run.smithi107.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-04-24T23:32:27.326 INFO:teuthology.orchestra.run.smithi195.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-24T23:32:27.343 INFO:teuthology.orchestra.run.smithi195.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 35 kB 00:00 2024-04-24T23:32:27.393 INFO:teuthology.orchestra.run.smithi195.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.3 MB/s | 3.0 MB 00:00 2024-04-24T23:32:27.402 INFO:teuthology.orchestra.run.smithi107.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-24T23:32:27.494 INFO:teuthology.orchestra.run.smithi195.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-04-24T23:32:27.494 INFO:teuthology.orchestra.run.smithi170.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.5 MB/s | 3.5 MB 00:00 2024-04-24T23:32:27.519 INFO:teuthology.orchestra.run.smithi107.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-24T23:32:27.535 INFO:teuthology.orchestra.run.smithi195.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-24T23:32:27.553 INFO:teuthology.orchestra.run.smithi170.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 4.2 MB/s | 249 kB 00:00 2024-04-24T23:32:27.578 INFO:teuthology.orchestra.run.smithi195.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-24T23:32:27.603 INFO:teuthology.orchestra.run.smithi195.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-24T23:32:27.619 INFO:teuthology.orchestra.run.smithi107.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-04-24T23:32:27.620 INFO:teuthology.orchestra.run.smithi170.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 739 kB/s | 49 kB 00:00 2024-04-24T23:32:27.669 INFO:teuthology.orchestra.run.smithi195.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-24T23:32:27.687 INFO:teuthology.orchestra.run.smithi170.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 229 kB/s | 15 kB 00:00 2024-04-24T23:32:27.694 INFO:teuthology.orchestra.run.smithi107.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.4 MB/s | 3.0 MB 00:00 2024-04-24T23:32:27.703 INFO:teuthology.orchestra.run.smithi195.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-24T23:32:27.728 INFO:teuthology.orchestra.run.smithi195.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-24T23:32:27.762 INFO:teuthology.orchestra.run.smithi170.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 264 kB 00:00 2024-04-24T23:32:27.770 INFO:teuthology.orchestra.run.smithi195.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-04-24T23:32:27.846 INFO:teuthology.orchestra.run.smithi195.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-24T23:32:27.871 INFO:teuthology.orchestra.run.smithi195.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-24T23:32:27.878 INFO:teuthology.orchestra.run.smithi107.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 582 kB 00:00 2024-04-24T23:32:27.887 INFO:teuthology.orchestra.run.smithi170.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.0 MB/s | 762 kB 00:00 2024-04-24T23:32:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-04-24T23:32:27.921 INFO:teuthology.orchestra.run.smithi195.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-24T23:32:27.928 INFO:teuthology.orchestra.run.smithi107.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-04-24T23:32:27.947 INFO:teuthology.orchestra.run.smithi195.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-24T23:32:27.962 INFO:teuthology.orchestra.run.smithi170.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 244 kB/s | 18 kB 00:00 2024-04-24T23:32:27.972 INFO:teuthology.orchestra.run.smithi195.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-04-24T23:32:27.979 INFO:teuthology.orchestra.run.smithi107.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-24T23:32:28.029 INFO:teuthology.orchestra.run.smithi107.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-04-24T23:32:28.049 INFO:teuthology.orchestra.run.smithi195.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-24T23:32:28.074 INFO:teuthology.orchestra.run.smithi195.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-24T23:32:28.088 INFO:teuthology.orchestra.run.smithi107.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-24T23:32:28.100 INFO:teuthology.orchestra.run.smithi195.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-24T23:32:28.125 INFO:teuthology.orchestra.run.smithi195.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-24T23:32:28.138 INFO:teuthology.orchestra.run.smithi107.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-24T23:32:28.150 INFO:teuthology.orchestra.run.smithi195.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-24T23:32:28.179 INFO:teuthology.orchestra.run.smithi195.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-24T23:32:28.188 INFO:teuthology.orchestra.run.smithi107.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-04-24T23:32:28.201 INFO:teuthology.orchestra.run.smithi195.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-24T23:32:28.226 INFO:teuthology.orchestra.run.smithi195.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-24T23:32:28.251 INFO:teuthology.orchestra.run.smithi195.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-24T23:32:28.277 INFO:teuthology.orchestra.run.smithi195.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-24T23:32:28.288 INFO:teuthology.orchestra.run.smithi170.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 6.8 MB/s | 7.4 MB 00:01 2024-04-24T23:32:28.352 INFO:teuthology.orchestra.run.smithi195.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-24T23:32:28.377 INFO:teuthology.orchestra.run.smithi195.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-24T23:32:28.403 INFO:teuthology.orchestra.run.smithi195.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-04-24T23:32:28.405 INFO:teuthology.orchestra.run.smithi170.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 182 kB/s | 80 kB 00:00 2024-04-24T23:32:28.428 INFO:teuthology.orchestra.run.smithi195.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-24T23:32:28.445 INFO:teuthology.orchestra.run.smithi195.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 860 kB/s | 36 kB 00:00 2024-04-24T23:32:28.461 INFO:teuthology.orchestra.run.smithi195.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-24T23:32:28.487 INFO:teuthology.orchestra.run.smithi195.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-04-24T23:32:28.504 INFO:teuthology.orchestra.run.smithi195.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2024-04-24T23:32:28.529 INFO:teuthology.orchestra.run.smithi195.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-24T23:32:28.546 INFO:teuthology.orchestra.run.smithi195.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-24T23:32:28.563 INFO:teuthology.orchestra.run.smithi195.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-24T23:32:28.580 INFO:teuthology.orchestra.run.smithi195.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 664 kB/s | 22 kB 00:00 2024-04-24T23:32:28.614 INFO:teuthology.orchestra.run.smithi195.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 17 MB/s | 863 kB 00:00 2024-04-24T23:32:28.663 INFO:teuthology.orchestra.run.smithi170.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 757 kB/s | 195 kB 00:00 2024-04-24T23:32:28.814 INFO:teuthology.orchestra.run.smithi107.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-04-24T23:32:28.855 INFO:teuthology.orchestra.run.smithi170.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-04-24T23:32:28.872 INFO:teuthology.orchestra.run.smithi195.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-24T23:32:29.023 INFO:teuthology.orchestra.run.smithi195.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 809 kB/s | 309 kB 00:00 2024-04-24T23:32:29.048 INFO:teuthology.orchestra.run.smithi195.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-24T23:32:29.073 INFO:teuthology.orchestra.run.smithi195.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-04-24T23:32:29.089 INFO:teuthology.orchestra.run.smithi170.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-24T23:32:29.106 INFO:teuthology.orchestra.run.smithi107.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-24T23:32:29.139 INFO:teuthology.orchestra.run.smithi170.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 4.3 MB/s | 1.2 MB 00:00 2024-04-24T23:32:29.197 INFO:teuthology.orchestra.run.smithi107.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 4.5 MB/s | 4.7 MB 00:01 2024-04-24T23:32:29.198 INFO:teuthology.orchestra.run.smithi170.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-04-24T23:32:29.240 INFO:teuthology.orchestra.run.smithi170.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-04-24T23:32:29.247 INFO:teuthology.orchestra.run.smithi107.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-04-24T23:32:29.306 INFO:teuthology.orchestra.run.smithi107.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-24T23:32:29.307 INFO:teuthology.orchestra.run.smithi170.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-24T23:32:29.349 INFO:teuthology.orchestra.run.smithi170.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-04-24T23:32:29.432 INFO:teuthology.orchestra.run.smithi170.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 5.4 MB/s | 689 kB 00:00 2024-04-24T23:32:29.466 INFO:teuthology.orchestra.run.smithi170.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 7.9 MB/s | 943 kB 00:00 2024-04-24T23:32:29.581 INFO:teuthology.orchestra.run.smithi107.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-04-24T23:32:29.630 INFO:teuthology.orchestra.run.smithi170.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 541 kB/s | 107 kB 00:00 2024-04-24T23:32:29.650 INFO:teuthology.orchestra.run.smithi170.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 117 kB/s | 21 kB 00:00 2024-04-24T23:32:29.673 INFO:teuthology.orchestra.run.smithi195.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-04-24T23:32:29.690 INFO:teuthology.orchestra.run.smithi107.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 809 kB/s | 87 kB 00:00 2024-04-24T23:32:29.698 INFO:teuthology.orchestra.run.smithi195.stdout:(109/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 40 kB/s | 26 kB 00:00 2024-04-24T23:32:29.724 INFO:teuthology.orchestra.run.smithi195.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 75 kB/s | 49 kB 00:00 2024-04-24T23:32:29.749 INFO:teuthology.orchestra.run.smithi195.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-24T23:32:29.750 INFO:teuthology.orchestra.run.smithi170.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-24T23:32:29.774 INFO:teuthology.orchestra.run.smithi195.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-24T23:32:29.776 INFO:teuthology.orchestra.run.smithi170.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-04-24T23:32:29.858 INFO:teuthology.orchestra.run.smithi195.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-04-24T23:32:29.884 INFO:teuthology.orchestra.run.smithi170.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-04-24T23:32:29.917 INFO:teuthology.orchestra.run.smithi195.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-04-24T23:32:29.967 INFO:teuthology.orchestra.run.smithi195.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 227 kB/s | 43 kB 00:00 2024-04-24T23:32:29.984 INFO:teuthology.orchestra.run.smithi170.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-24T23:32:29.992 INFO:teuthology.orchestra.run.smithi195.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 242 kB/s | 32 kB 00:00 2024-04-24T23:32:30.018 INFO:teuthology.orchestra.run.smithi195.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-24T23:32:30.051 INFO:teuthology.orchestra.run.smithi195.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-24T23:32:30.051 INFO:teuthology.orchestra.run.smithi170.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.9 MB/s | 3.0 MB 00:00 2024-04-24T23:32:30.076 INFO:teuthology.orchestra.run.smithi195.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-24T23:32:30.090 INFO:teuthology.orchestra.run.smithi107.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 6.9 MB/s | 5.8 MB 00:00 2024-04-24T23:32:30.093 INFO:teuthology.orchestra.run.smithi170.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-24T23:32:30.101 INFO:teuthology.orchestra.run.smithi195.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-24T23:32:30.127 INFO:teuthology.orchestra.run.smithi195.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 474 kB/s | 35 kB 00:00 2024-04-24T23:32:30.152 INFO:teuthology.orchestra.run.smithi195.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-24T23:32:30.152 INFO:teuthology.orchestra.run.smithi170.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-04-24T23:32:30.199 INFO:teuthology.orchestra.run.smithi107.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-24T23:32:30.202 INFO:teuthology.orchestra.run.smithi170.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-24T23:32:30.294 INFO:teuthology.orchestra.run.smithi195.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-24T23:32:30.303 INFO:teuthology.orchestra.run.smithi170.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-24T23:32:30.316 INFO:teuthology.orchestra.run.smithi107.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-24T23:32:30.319 INFO:teuthology.orchestra.run.smithi195.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 755 kB/s | 144 kB 00:00 2024-04-24T23:32:30.344 INFO:teuthology.orchestra.run.smithi195.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 56 kB/s | 11 kB 00:00 2024-04-24T23:32:30.370 INFO:teuthology.orchestra.run.smithi195.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-24T23:32:30.395 INFO:teuthology.orchestra.run.smithi195.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-24T23:32:30.411 INFO:teuthology.orchestra.run.smithi170.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-24T23:32:30.420 INFO:teuthology.orchestra.run.smithi195.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-04-24T23:32:30.424 INFO:teuthology.orchestra.run.smithi107.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 237 kB/s | 26 kB 00:00 2024-04-24T23:32:30.445 INFO:teuthology.orchestra.run.smithi195.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-04-24T23:32:30.504 INFO:teuthology.orchestra.run.smithi195.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-24T23:32:30.520 INFO:teuthology.orchestra.run.smithi170.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-24T23:32:30.529 INFO:teuthology.orchestra.run.smithi195.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-04-24T23:32:30.533 INFO:teuthology.orchestra.run.smithi107.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-04-24T23:32:30.554 INFO:teuthology.orchestra.run.smithi195.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 730 kB/s | 79 kB 00:00 2024-04-24T23:32:30.579 INFO:teuthology.orchestra.run.smithi195.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-04-24T23:32:30.605 INFO:teuthology.orchestra.run.smithi195.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-24T23:32:30.628 INFO:teuthology.orchestra.run.smithi170.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-24T23:32:30.630 INFO:teuthology.orchestra.run.smithi195.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-24T23:32:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-24T23:32:30.655 INFO:teuthology.orchestra.run.smithi195.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-24T23:32:30.680 INFO:teuthology.orchestra.run.smithi195.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 567 kB/s | 28 kB 00:00 2024-04-24T23:32:30.706 INFO:teuthology.orchestra.run.smithi195.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-04-24T23:32:30.729 INFO:teuthology.orchestra.run.smithi170.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-04-24T23:32:30.731 INFO:teuthology.orchestra.run.smithi195.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-24T23:32:30.742 INFO:teuthology.orchestra.run.smithi107.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-04-24T23:32:30.756 INFO:teuthology.orchestra.run.smithi195.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-24T23:32:30.781 INFO:teuthology.orchestra.run.smithi195.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-04-24T23:32:30.806 INFO:teuthology.orchestra.run.smithi195.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 680 kB/s | 51 kB 00:00 2024-04-24T23:32:30.826 INFO:teuthology.orchestra.run.smithi107.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2024-04-24T23:32:30.832 INFO:teuthology.orchestra.run.smithi195.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-24T23:32:30.857 INFO:teuthology.orchestra.run.smithi195.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-24T23:32:30.891 INFO:teuthology.orchestra.run.smithi195.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-24T23:32:30.908 INFO:teuthology.orchestra.run.smithi195.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-04-24T23:32:30.933 INFO:teuthology.orchestra.run.smithi195.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-24T23:32:30.934 INFO:teuthology.orchestra.run.smithi107.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-24T23:32:31.025 INFO:teuthology.orchestra.run.smithi195.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-24T23:32:31.034 INFO:teuthology.orchestra.run.smithi107.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-24T23:32:31.087 INFO:teuthology.orchestra.run.smithi170.stdout:(65/152): libtool-2.4.6-45.el9.x86_64.rpm 623 kB/s | 582 kB 00:00 2024-04-24T23:32:31.135 INFO:teuthology.orchestra.run.smithi107.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 772 kB/s | 77 kB 00:00 2024-04-24T23:32:31.184 INFO:teuthology.orchestra.run.smithi195.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-04-24T23:32:31.227 INFO:teuthology.orchestra.run.smithi107.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 461 kB/s | 42 kB 00:00 2024-04-24T23:32:31.267 INFO:teuthology.orchestra.run.smithi195.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-24T23:32:31.335 INFO:teuthology.orchestra.run.smithi107.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-24T23:32:31.402 INFO:teuthology.orchestra.run.smithi107.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.7 MB/s | 5.1 MB 00:00 2024-04-24T23:32:31.451 INFO:teuthology.orchestra.run.smithi195.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-04-24T23:32:31.819 INFO:teuthology.orchestra.run.smithi107.stdout:(87/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 4.4 MB/s | 48 MB 00:11 2024-04-24T23:32:31.845 INFO:teuthology.orchestra.run.smithi107.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 312 kB/s | 159 kB 00:00 2024-04-24T23:32:31.870 INFO:teuthology.orchestra.run.smithi107.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 597 kB/s | 279 kB 00:00 2024-04-24T23:32:31.945 INFO:teuthology.orchestra.run.smithi107.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 536 kB/s | 54 kB 00:00 2024-04-24T23:32:31.984 INFO:teuthology.orchestra.run.smithi195.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.1 MB/s | 3.4 MB 00:01 2024-04-24T23:32:31.989 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:32:31.989 INFO:teuthology.orchestra.run.smithi195.stdout:Total 13 MB/s | 220 MB 00:17 2024-04-24T23:32:32.045 INFO:teuthology.orchestra.run.smithi107.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-24T23:32:32.146 INFO:teuthology.orchestra.run.smithi107.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-24T23:32:32.162 INFO:teuthology.orchestra.run.smithi107.stdout:(93/152): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-04-24T23:32:32.254 INFO:teuthology.orchestra.run.smithi107.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-24T23:32:32.271 INFO:teuthology.orchestra.run.smithi107.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 331 kB/s | 36 kB 00:00 2024-04-24T23:32:32.355 INFO:teuthology.orchestra.run.smithi107.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-04-24T23:32:32.380 INFO:teuthology.orchestra.run.smithi170.stdout:(66/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 7.2 MB/s | 48 MB 00:06 2024-04-24T23:32:32.455 INFO:teuthology.orchestra.run.smithi107.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 579 kB/s | 58 kB 00:00 2024-04-24T23:32:32.555 INFO:teuthology.orchestra.run.smithi107.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 938 kB/s | 94 kB 00:00 2024-04-24T23:32:32.614 INFO:teuthology.orchestra.run.smithi107.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 890 kB/s | 305 kB 00:00 2024-04-24T23:32:32.647 INFO:teuthology.orchestra.run.smithi107.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 693 kB/s | 64 kB 00:00 2024-04-24T23:32:32.706 INFO:teuthology.orchestra.run.smithi107.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 488 kB/s | 45 kB 00:00 2024-04-24T23:32:32.807 INFO:teuthology.orchestra.run.smithi107.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-04-24T23:32:33.049 INFO:teuthology.orchestra.run.smithi107.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-04-24T23:32:33.073 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-04-24T23:32:33.132 INFO:teuthology.orchestra.run.smithi107.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-04-24T23:32:33.174 INFO:teuthology.orchestra.run.smithi107.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-24T23:32:33.191 INFO:teuthology.orchestra.run.smithi107.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-04-24T23:32:33.216 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-04-24T23:32:33.216 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-04-24T23:32:33.235 INFO:teuthology.orchestra.run.smithi107.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 595 kB/s | 26 kB 00:00 2024-04-24T23:32:33.277 INFO:teuthology.orchestra.run.smithi107.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-24T23:32:33.305 INFO:teuthology.orchestra.run.smithi170.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.7 MB/s | 1.6 MB 00:00 2024-04-24T23:32:33.319 INFO:teuthology.orchestra.run.smithi107.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-24T23:32:33.378 INFO:teuthology.orchestra.run.smithi107.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-24T23:32:33.519 INFO:teuthology.orchestra.run.smithi107.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-24T23:32:33.582 INFO:teuthology.orchestra.run.smithi170.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-04-24T23:32:33.603 INFO:teuthology.orchestra.run.smithi107.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-24T23:32:33.637 INFO:teuthology.orchestra.run.smithi107.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-24T23:32:33.670 INFO:teuthology.orchestra.run.smithi107.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 968 kB/s | 32 kB 00:00 2024-04-24T23:32:33.690 INFO:teuthology.orchestra.run.smithi170.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-24T23:32:33.754 INFO:teuthology.orchestra.run.smithi107.stdout:(115/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-04-24T23:32:33.771 INFO:teuthology.orchestra.run.smithi107.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-24T23:32:33.813 INFO:teuthology.orchestra.run.smithi107.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-24T23:32:33.838 INFO:teuthology.orchestra.run.smithi107.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-24T23:32:33.880 INFO:teuthology.orchestra.run.smithi107.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 846 kB/s | 35 kB 00:00 2024-04-24T23:32:33.905 INFO:teuthology.orchestra.run.smithi107.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-24T23:32:33.931 INFO:teuthology.orchestra.run.smithi107.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-24T23:32:33.981 INFO:teuthology.orchestra.run.smithi107.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-24T23:32:34.015 INFO:teuthology.orchestra.run.smithi107.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 321 kB/s | 11 kB 00:00 2024-04-24T23:32:34.173 INFO:teuthology.orchestra.run.smithi107.stdout:(124/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-04-24T23:32:34.190 INFO:teuthology.orchestra.run.smithi107.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 101 kB/s | 18 kB 00:00 2024-04-24T23:32:34.223 INFO:teuthology.orchestra.run.smithi107.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 586 kB/s | 19 kB 00:00 2024-04-24T23:32:34.257 INFO:teuthology.orchestra.run.smithi107.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-04-24T23:32:34.274 INFO:teuthology.orchestra.run.smithi107.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-24T23:32:34.308 INFO:teuthology.orchestra.run.smithi107.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-04-24T23:32:34.358 INFO:teuthology.orchestra.run.smithi107.stdout:(130/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.5 MB/s | 2.0 MB 00:00 2024-04-24T23:32:34.375 INFO:teuthology.orchestra.run.smithi107.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-24T23:32:34.392 INFO:teuthology.orchestra.run.smithi107.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-04-24T23:32:34.425 INFO:teuthology.orchestra.run.smithi107.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-24T23:32:34.442 INFO:teuthology.orchestra.run.smithi107.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-24T23:32:34.468 INFO:teuthology.orchestra.run.smithi107.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-24T23:32:34.485 INFO:teuthology.orchestra.run.smithi107.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 673 kB/s | 28 kB 00:00 2024-04-24T23:32:34.527 INFO:teuthology.orchestra.run.smithi107.stdout:(137/152): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-04-24T23:32:34.544 INFO:teuthology.orchestra.run.smithi107.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 406 kB/s | 31 kB 00:00 2024-04-24T23:32:34.569 INFO:teuthology.orchestra.run.smithi107.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-24T23:32:34.594 INFO:teuthology.orchestra.run.smithi107.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-24T23:32:34.620 INFO:teuthology.orchestra.run.smithi107.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 639 kB/s | 59 kB 00:00 2024-04-24T23:32:34.645 INFO:teuthology.orchestra.run.smithi107.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-24T23:32:34.670 INFO:teuthology.orchestra.run.smithi107.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-24T23:32:34.696 INFO:teuthology.orchestra.run.smithi107.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-24T23:32:34.721 INFO:teuthology.orchestra.run.smithi107.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-24T23:32:34.747 INFO:teuthology.orchestra.run.smithi107.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-24T23:32:34.779 INFO:teuthology.orchestra.run.smithi107.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-24T23:32:34.863 INFO:teuthology.orchestra.run.smithi107.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-24T23:32:34.956 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-04-24T23:32:34.957 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-04-24T23:32:35.055 INFO:teuthology.orchestra.run.smithi107.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 1.3 MB 00:00 2024-04-24T23:32:35.066 INFO:teuthology.orchestra.run.smithi170.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 4.2 MB/s | 5.8 MB 00:01 2024-04-24T23:32:35.139 INFO:teuthology.orchestra.run.smithi107.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-24T23:32:35.255 INFO:teuthology.orchestra.run.smithi107.stdout:(151/152): librados2-19.0.0-2445.gd2c45d97.el9. 6.7 MB/s | 3.4 MB 00:00 2024-04-24T23:32:35.291 INFO:teuthology.orchestra.run.smithi170.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.0 MB/s | 4.7 MB 00:04 2024-04-24T23:32:35.306 INFO:teuthology.orchestra.run.smithi107.stdout:(152/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-04-24T23:32:35.308 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:32:35.309 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 220 MB 00:19 2024-04-24T23:32:35.383 INFO:teuthology.orchestra.run.smithi170.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-04-24T23:32:35.483 INFO:teuthology.orchestra.run.smithi170.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 456 kB/s | 87 kB 00:00 2024-04-24T23:32:35.850 INFO:teuthology.orchestra.run.smithi170.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 666 kB/s | 244 kB 00:00 2024-04-24T23:32:36.192 INFO:teuthology.orchestra.run.smithi170.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 726 kB/s | 248 kB 00:00 2024-04-24T23:32:36.292 INFO:teuthology.orchestra.run.smithi170.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-24T23:32:36.392 INFO:teuthology.orchestra.run.smithi170.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-04-24T23:32:36.449 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-24T23:32:36.596 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-24T23:32:36.596 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-24T23:32:36.676 INFO:teuthology.orchestra.run.smithi170.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-04-24T23:32:36.701 INFO:teuthology.orchestra.run.smithi170.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 558 kB/s | 172 kB 00:00 2024-04-24T23:32:36.785 INFO:teuthology.orchestra.run.smithi170.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-04-24T23:32:36.885 INFO:teuthology.orchestra.run.smithi170.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-04-24T23:32:36.994 INFO:teuthology.orchestra.run.smithi170.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-24T23:32:37.110 INFO:teuthology.orchestra.run.smithi170.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 662 kB/s | 77 kB 00:00 2024-04-24T23:32:37.211 INFO:teuthology.orchestra.run.smithi170.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 422 kB/s | 42 kB 00:00 2024-04-24T23:32:37.319 INFO:teuthology.orchestra.run.smithi170.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-24T23:32:37.520 INFO:teuthology.orchestra.run.smithi170.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 794 kB/s | 159 kB 00:00 2024-04-24T23:32:37.628 INFO:teuthology.orchestra.run.smithi170.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-24T23:32:37.729 INFO:teuthology.orchestra.run.smithi170.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 510 kB/s | 51 kB 00:00 2024-04-24T23:32:37.796 INFO:teuthology.orchestra.run.smithi170.stdout:(89/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 675 kB/s | 4.4 MB 00:06 2024-04-24T23:32:37.838 INFO:teuthology.orchestra.run.smithi170.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 494 kB/s | 54 kB 00:00 2024-04-24T23:32:37.938 INFO:teuthology.orchestra.run.smithi170.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-24T23:32:38.047 INFO:teuthology.orchestra.run.smithi170.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 407 kB/s | 44 kB 00:00 2024-04-24T23:32:38.364 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-24T23:32:38.364 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-24T23:32:38.397 INFO:teuthology.orchestra.run.smithi170.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 631 kB/s | 221 kB 00:00 2024-04-24T23:32:38.497 INFO:teuthology.orchestra.run.smithi170.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 359 kB/s | 36 kB 00:00 2024-04-24T23:32:38.606 INFO:teuthology.orchestra.run.smithi170.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-04-24T23:32:39.039 INFO:teuthology.orchestra.run.smithi170.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 704 kB/s | 305 kB 00:00 2024-04-24T23:32:39.156 INFO:teuthology.orchestra.run.smithi170.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 494 kB/s | 58 kB 00:00 2024-04-24T23:32:39.365 INFO:teuthology.orchestra.run.smithi170.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 450 kB/s | 94 kB 00:00 2024-04-24T23:32:39.549 INFO:teuthology.orchestra.run.smithi170.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 347 kB/s | 64 kB 00:00 2024-04-24T23:32:39.657 INFO:teuthology.orchestra.run.smithi170.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 414 kB/s | 45 kB 00:00 2024-04-24T23:32:39.766 INFO:teuthology.orchestra.run.smithi170.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-04-24T23:32:40.583 INFO:teuthology.orchestra.run.smithi170.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.8 MB/s | 16 MB 00:02 2024-04-24T23:32:40.866 INFO:teuthology.orchestra.run.smithi170.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-24T23:32:40.892 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-04-24T23:32:40.933 INFO:teuthology.orchestra.run.smithi170.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-04-24T23:32:40.979 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:32:40.983 INFO:teuthology.orchestra.run.smithi170.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 709 kB/s | 863 kB 00:01 2024-04-24T23:32:41.009 INFO:teuthology.orchestra.run.smithi170.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-04-24T23:32:41.067 INFO:teuthology.orchestra.run.smithi170.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-24T23:32:41.079 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:32:41.126 INFO:teuthology.orchestra.run.smithi170.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 837 kB/s | 49 kB 00:00 2024-04-24T23:32:41.176 INFO:teuthology.orchestra.run.smithi170.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-24T23:32:41.243 INFO:teuthology.orchestra.run.smithi170.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-24T23:32:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:32:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:32:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:32:41.343 INFO:teuthology.orchestra.run.smithi170.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-24T23:32:41.382 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:32:41.427 INFO:teuthology.orchestra.run.smithi170.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-24T23:32:41.495 INFO:teuthology.orchestra.run.smithi170.stdout:(113/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-04-24T23:32:41.519 INFO:teuthology.orchestra.run.smithi170.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-04-24T23:32:41.553 INFO:teuthology.orchestra.run.smithi170.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-04-24T23:32:41.578 INFO:teuthology.orchestra.run.smithi170.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-04-24T23:32:41.612 INFO:teuthology.orchestra.run.smithi170.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-24T23:32:41.637 INFO:teuthology.orchestra.run.smithi170.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-24T23:32:41.671 INFO:teuthology.orchestra.run.smithi170.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-24T23:32:41.696 INFO:teuthology.orchestra.run.smithi170.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-04-24T23:32:41.712 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:32:41.729 INFO:teuthology.orchestra.run.smithi170.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-24T23:32:41.805 INFO:teuthology.orchestra.run.smithi170.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-24T23:32:41.830 INFO:teuthology.orchestra.run.smithi170.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-24T23:32:41.840 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:41.864 INFO:teuthology.orchestra.run.smithi170.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-04-24T23:32:41.889 INFO:teuthology.orchestra.run.smithi170.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 300 kB/s | 18 kB 00:00 2024-04-24T23:32:41.923 INFO:teuthology.orchestra.run.smithi170.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-04-24T23:32:41.948 INFO:teuthology.orchestra.run.smithi170.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-04-24T23:32:41.961 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:41.982 INFO:teuthology.orchestra.run.smithi170.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-04-24T23:32:41.991 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:42.032 INFO:teuthology.orchestra.run.smithi170.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-24T23:32:42.057 INFO:teuthology.orchestra.run.smithi170.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-24T23:32:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:42.091 INFO:teuthology.orchestra.run.smithi170.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-24T23:32:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:32:42.116 INFO:teuthology.orchestra.run.smithi170.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-04-24T23:32:42.158 INFO:teuthology.orchestra.run.smithi170.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-24T23:32:42.183 INFO:teuthology.orchestra.run.smithi170.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-24T23:32:42.225 INFO:teuthology.orchestra.run.smithi170.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-24T23:32:42.251 INFO:teuthology.orchestra.run.smithi170.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-04-24T23:32:42.258 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:32:42.284 INFO:teuthology.orchestra.run.smithi170.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-04-24T23:32:42.309 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:32:42.310 INFO:teuthology.orchestra.run.smithi170.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-24T23:32:42.337 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:42.343 INFO:teuthology.orchestra.run.smithi170.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-24T23:32:42.369 INFO:teuthology.orchestra.run.smithi170.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-04-24T23:32:42.410 INFO:teuthology.orchestra.run.smithi170.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 761 kB/s | 51 kB 00:00 2024-04-24T23:32:42.439 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:42.444 INFO:teuthology.orchestra.run.smithi170.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-24T23:32:42.463 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:42.469 INFO:teuthology.orchestra.run.smithi170.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-24T23:32:42.525 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:42.548 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:42.565 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:42.595 INFO:teuthology.orchestra.run.smithi170.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-04-24T23:32:42.628 INFO:teuthology.orchestra.run.smithi170.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 126 kB/s | 20 kB 00:00 2024-04-24T23:32:42.654 INFO:teuthology.orchestra.run.smithi170.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-24T23:32:42.721 INFO:teuthology.orchestra.run.smithi170.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-24T23:32:42.747 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:42.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:42.888 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:32:42.981 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:32:43.013 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:32:43.047 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:32:43.129 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:32:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:32:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:32:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:32:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:32:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:32:43.397 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:32:43.429 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:32:43.464 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:32:43.521 INFO:teuthology.orchestra.run.smithi170.stdout:(148/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.9 MB/s | 3.4 MB 00:00 2024-04-24T23:32:43.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:32:43.580 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:32:43.618 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:32:43.639 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:32:43.707 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:32:43.713 INFO:teuthology.orchestra.run.smithi170.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-04-24T23:32:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:32:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:32:43.863 INFO:teuthology.orchestra.run.smithi170.stdout:(150/152): python3-numpy-1.20.1-5.el9.x86_64.rp 729 kB/s | 5.1 MB 00:07 2024-04-24T23:32:43.959 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:32:44.014 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:32:44.313 INFO:teuthology.orchestra.run.smithi170.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 1.3 MB 00:00 2024-04-24T23:32:44.327 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-24T23:32:44.442 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:32:44.569 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:32:44.661 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:32:44.690 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:32:44.699 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:32:44.732 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:32:44.754 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:32:44.763 INFO:teuthology.orchestra.run.smithi170.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.3 MB/s | 2.4 MB 00:01 2024-04-24T23:32:44.766 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:32:44.767 INFO:teuthology.orchestra.run.smithi170.stdout:Total 8.3 MB/s | 220 MB 00:26 2024-04-24T23:32:44.770 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:32:44.803 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:32:44.820 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:32:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:32:44.871 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:32:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:32:45.039 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:32:45.094 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:32:45.158 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:32:45.190 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:32:45.227 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:32:45.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:32:45.273 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:32:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:32:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:32:45.394 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:32:45.453 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:45.527 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:32:45.576 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:45.584 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:32:45.612 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:45.613 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:32:45.655 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:32:45.703 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:45.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:32:45.747 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:32:45.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:32:45.820 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:32:45.846 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:32:45.870 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:32:45.875 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-04-24T23:32:45.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:32:45.929 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:32:45.935 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:32:45.958 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:46.019 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-04-24T23:32:46.019 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-04-24T23:32:46.062 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:46.070 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:32:46.084 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:46.103 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:32:46.131 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:32:46.146 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:46.165 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:32:46.168 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:46.190 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:46.204 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:32:46.235 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:32:46.266 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:32:46.300 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:32:46.334 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:32:46.352 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:46.374 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:32:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:46.409 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:32:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:32:46.494 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:32:46.539 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:32:46.571 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:32:46.591 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:32:46.618 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:32:46.642 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:32:46.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:32:46.740 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:32:46.784 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:32:46.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:32:46.860 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:32:46.861 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:32:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:32:46.914 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:32:46.956 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:32:47.010 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:32:47.041 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:32:47.076 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:32:47.153 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:32:47.188 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:32:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:32:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:32:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:32:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:32:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:32:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:32:47.635 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:32:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:32:47.678 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:32:47.712 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:32:47.809 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:32:47.811 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-04-24T23:32:47.814 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-04-24T23:32:47.869 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:32:47.956 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:32:47.989 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:32:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:32:48.167 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:32:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:32:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:32:48.310 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:32:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:32:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:32:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:32:48.445 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:32:48.537 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:32:48.601 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:32:48.646 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:32:48.711 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:32:48.743 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:32:48.779 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:32:48.826 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:32:48.862 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:32:48.905 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:32:48.947 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:32:49.082 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:32:49.128 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:32:49.131 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:32:49.150 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:32:49.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:32:49.242 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:32:49.284 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:32:49.350 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:32:49.390 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:32:49.416 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:32:49.440 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:32:49.515 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:32:49.640 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:32:49.673 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:32:49.701 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:32:49.735 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:32:49.774 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:32:49.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:32:49.844 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:32:49.878 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:32:49.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:32:49.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:32:49.987 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:32:50.073 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:32:50.118 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:32:50.149 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:32:50.195 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:32:50.293 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:32:50.342 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:32:50.442 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:32:50.524 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:32:50.551 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:32:50.585 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:32:50.895 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:32:50.940 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:32:51.013 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:32:51.045 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:32:51.054 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:32:51.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:32:51.114 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:32:51.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:32:51.238 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:32:51.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:32:51.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:32:51.417 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:32:51.583 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:32:51.595 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:32:51.618 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:32:51.648 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:32:51.688 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:32:51.711 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:32:51.750 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:32:52.384 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:32:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:32:53.702 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:32:53.768 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:32:53.781 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-04-24T23:32:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:32:53.829 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:32:53.901 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:32:53.947 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:32:53.980 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:54.001 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:32:54.024 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:54.034 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:32:54.060 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:32:54.102 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:32:54.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:32:54.176 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:32:54.250 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:32:54.281 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:32:54.309 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:32:54.429 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:32:54.544 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:54.560 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:54.628 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:32:54.666 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:32:54.695 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:54.721 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:32:54.721 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:32:54.721 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:32:54.721 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:32:54.753 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:32:54.778 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:32:54.795 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:32:54.814 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:32:54.833 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:32:54.883 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:32:54.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:32:54.982 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:32:55.030 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:32:55.058 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:55.162 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:32:55.184 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:55.247 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:32:55.268 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:55.292 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:32:55.451 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:55.484 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:32:55.592 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:32:55.678 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:32:55.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:32:55.744 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:32:55.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:32:55.842 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:32:55.886 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:32:55.918 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:32:55.946 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:32:56.015 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:32:56.068 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:32:56.108 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:32:56.143 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:32:56.219 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:32:56.254 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:32:56.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:32:56.326 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:32:56.396 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:32:56.441 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:32:56.516 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:32:56.655 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:32:56.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:32:57.211 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:57.220 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:32:57.243 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:32:57.343 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:32:57.428 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:32:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:32:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:32:57.672 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:32:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:32:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:32:57.784 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:32:57.822 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:32:57.845 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:32:57.886 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:32:57.913 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:32:57.913 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:32:57.913 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:32:57.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:32:57.931 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:32:57.939 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:32:57.978 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:32:58.014 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:32:58.056 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:32:58.098 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:32:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:32:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:32:58.301 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:32:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:32:58.463 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:32:58.644 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:32:58.683 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:32:58.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:32:58.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:32:58.808 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:32:58.933 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:32:58.966 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:32:59.002 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:32:59.037 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:32:59.076 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:32:59.107 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:32:59.138 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:32:59.172 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:32:59.206 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:32:59.247 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:32:59.282 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:32:59.368 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:32:59.411 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:32:59.461 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:32:59.506 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:32:59.604 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:32:59.653 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:33:00.358 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:33:00.409 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:33:00.442 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:33:00.539 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:33:00.591 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:33:00.669 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:33:00.711 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:33:00.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:33:00.917 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:33:00.995 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:33:01.626 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:33:01.747 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:33:02.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:33:03.056 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:33:03.083 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:33:03.116 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:33:03.426 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:33:03.471 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:33:03.805 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:33:03.834 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:33:03.872 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:33:04.341 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:33:04.374 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:33:04.446 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:33:04.476 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:33:06.450 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:33:06.450 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-04-24T23:33:06.450 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-04-24T23:33:06.450 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-04-24T23:33:06.451 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-04-24T23:33:06.451 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-04-24T23:33:06.451 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-04-24T23:33:06.451 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-04-24T23:33:06.451 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:06.616 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:06.631 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:06.632 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:06.632 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:33:06.632 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:06.745 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:33:06.781 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:33:07.316 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:33:07.359 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:33:07.451 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:33:07.451 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:33:07.451 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:33:07.451 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:07.477 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:33:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:08.144 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:08.144 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:08.144 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:33:08.145 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:08.145 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:08.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:08.370 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:08.392 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:08.828 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:08.851 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:08.958 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:33:09.007 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:33:09.062 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:33:09.247 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:33:09.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:33:09.375 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:33:09.461 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-04-24T23:33:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-04-24T23:33:09.574 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-04-24T23:33:09.574 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:09.592 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:09.629 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:33:09.661 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:33:09.689 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:33:09.752 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:09.768 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:09.768 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:09.768 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:33:09.768 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:10.080 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:10.112 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:10.144 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:33:10.207 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:33:10.235 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:11.238 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:11.263 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:11.264 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:11.264 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:11.264 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:33:11.264 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:11.265 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:11.265 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:11.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:11.488 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:11.511 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:11.511 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:11.511 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:33:11.511 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:11.512 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:11.512 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:11.956 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:11.978 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:11.978 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:11.978 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:33:11.979 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:11.979 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:11.979 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:12.077 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:33:12.126 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:33:12.182 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:33:12.255 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:12.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:12.367 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:33:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:12.415 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:33:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:33:12.558 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:33:12.581 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:12.598 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:12.622 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:12.666 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:12.677 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:12.721 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:12.764 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:33:12.774 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:33:12.797 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:33:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:33:12.825 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:33:13.274 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:13.306 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:13.338 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:33:13.399 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:33:13.438 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:13.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:33:13.751 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:13.773 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:13.773 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:13.773 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:33:13.774 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:13.774 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:13.774 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:13.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:13.840 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:13.841 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:13.841 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:33:13.841 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:13.886 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:13.908 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:13.908 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:13.908 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:33:13.908 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:13.909 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:13.909 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:14.220 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:14.242 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:14.242 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:14.242 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:33:14.242 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:14.242 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:14.243 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:14.470 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:14.515 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:15.478 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:15.507 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:15.607 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:15.770 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:33:15.800 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:15.824 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:15.868 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:15.880 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:15.977 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:33:16.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:33:16.929 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:33:16.962 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:16.984 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:16.984 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:16.985 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:33:16.985 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:16.985 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:16.985 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:17.029 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:17.052 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:17.052 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:17.052 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:33:17.052 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:17.105 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:17.127 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:17.127 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:17.127 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:33:17.128 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:17.128 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:17.128 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:33:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:17.454 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:18.766 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:33:18.807 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:33:18.847 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:33:18.880 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:33:18.907 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-04-24T23:33:19.025 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-04-24T23:33:19.026 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-04-24T23:33:19.026 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:19.088 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:33:19.137 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:33:19.166 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:33:19.199 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:33:19.214 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:19.214 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:33:19.214 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:19.745 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:19.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:19.801 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:20.638 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:33:20.668 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:33:20.694 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:20.727 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:33:20.768 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:33:20.849 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:33:20.881 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:33:20.881 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:20.919 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:20.941 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:33:20.941 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:20.941 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:33:20.941 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:20.941 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:33:20.942 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:20.965 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:33:21.409 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:21.409 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:33:21.409 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:21.410 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:33:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:21.508 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:33:21.556 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:33:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:33:21.797 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:33:21.846 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:33:21.922 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:33:21.967 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:33:22.009 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:33:22.019 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:22.049 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:33:22.082 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:33:22.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:33:22.168 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:22.204 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:33:22.238 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:33:22.264 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:33:22.291 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:33:22.339 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:33:22.368 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:33:22.663 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:22.687 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:33:22.718 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:33:22.780 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:33:22.819 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:23.014 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:23.058 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:23.077 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:23.853 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:33:23.887 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:23.899 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:33:23.988 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:33:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:33:24.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:33:24.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:33:24.142 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:24.226 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:24.259 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:24.259 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:24.424 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:33:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:33:25.145 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:33:25.173 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:33:25.196 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:25.240 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:25.252 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:33:25.366 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:33:25.413 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:33:26.316 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:33:26.351 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:33:26.374 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:26.418 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:26.441 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:33:26.512 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:26.512 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:33:26.512 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:26.513 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:33:26.517 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:33:26.778 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:33:26.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:33:26.780 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:33:26.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:33:26.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:33:26.795 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:33:26.796 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:33:26.797 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:33:26.798 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:33:26.799 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:33:26.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:33:26.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:26.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:33:26.820 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:26.842 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:33:26.843 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:33:26.843 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:33:26.843 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:26.843 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:33:26.843 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:33:28.892 INFO:teuthology.orchestra.run.smithi195.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.893 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:33:28.894 INFO:teuthology.orchestra.run.smithi195.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:33:28.895 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:33:28.896 INFO:teuthology.orchestra.run.smithi195.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:33:28.897 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:33:28.898 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:33:28.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:33:28.900 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:33:28.901 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:33:28.902 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:33:28.903 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:33:28.904 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-04-24T23:33:29.292 DEBUG:teuthology.parallel:result is None 2024-04-24T23:33:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:33:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:33:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:33:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:33:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:33:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:33:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:33:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:33:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:33:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:33:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:33:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:33:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:33:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:33:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:33:31.382 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:33:31.422 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:33:31.454 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:33:31.482 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:33:31.663 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:33:31.714 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:33:31.733 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:33:32.245 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.246 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:33:32.248 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:33:32.249 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:33:32.250 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:33:32.251 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:33:32.252 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:32.253 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:33:32.254 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:33:32.255 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:33:32.256 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:33:32.257 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:33:32.257 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:33:32.257 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-24T23:33:32.322 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:32.364 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:32.384 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:33:32.700 DEBUG:teuthology.parallel:result is None 2024-04-24T23:33:33.212 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:33:33.303 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:33:33.343 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:33:33.424 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:33:33.457 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:33:33.457 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:33.541 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:33:33.574 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:33.574 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:33.669 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:33:33.740 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:33:39.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:33:39.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:33:39.904 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:33:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:33:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:33:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:33:39.920 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:33:39.921 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:33:39.922 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:33:39.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:33:39.924 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:33:39.925 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:33:39.926 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:33:39.927 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:33:39.927 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:33:39.927 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:33:42.013 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:33:42.013 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:42.013 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.014 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.015 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:33:42.016 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.017 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:33:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.019 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:33:42.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:33:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:33:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:33:42.023 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:33:42.024 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:33:42.025 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:33:42.025 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:33:42.025 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-04-24T23:33:42.025 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-04-24T23:33:42.399 DEBUG:teuthology.parallel:result is None 2024-04-24T23:33:42.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:42.637 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:33:42.663 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:42.663 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:42.663 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:33:42.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:42.886 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:33:42.908 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:42.908 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:42.908 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:33:42.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:43.050 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:33:43.077 INFO:teuthology.orchestra.run.smithi195.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:43.077 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:33:43.077 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:33:43.080 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T23:33:43.080 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:33:43.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:33:43.112 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:33:43.112 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:33:43.139 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:33:43.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:33:43.169 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T23:33:43.169 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:33:43.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:33:43.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:33:43.263 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:33:43.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:33:43.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:33:43.351 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:33:43.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:33:43.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:33:43.443 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T23:33:43.443 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:33:43.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:33:43.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:33:43.531 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:33:43.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:33:43.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:33:43.618 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:33:43.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:33:43.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:33:43.706 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T23:33:43.706 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:33:43.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:33:43.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:33:43.793 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:33:43.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:33:43.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:33:43.879 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:33:43.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:33:43.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:33:43.966 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-24T23:33:44.070 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2}} 2024-04-24T23:33:44.071 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:44.071 INFO:tasks.cephadm:Cluster fsid is 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:33:44.071 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-24T23:33:44.072 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.107', 'mon.b': '172.21.15.170', 'mon.c': '172.21.15.195'} 2024-04-24T23:33:44.072 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-04-24T23:33:44.072 INFO:tasks.cephadm:First mgr is x 2024-04-24T23:33:44.072 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-24T23:33:44.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-04-24T23:33:44.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-04-24T23:33:44.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-04-24T23:33:44.148 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-24T23:33:44.148 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:44.288 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-24T23:33:44.504 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:33:44.505 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-24T23:33:44.506 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:45.018 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 24 23:33 /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:45.019 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:45.540 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 24 23:33 /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:45.540 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:46.096 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 24 23:33 /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:46.097 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:46.115 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:46.134 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T23:33:46.169 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-24T23:33:46.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-24T23:33:46.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-24T23:33:46.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-24T23:33:47.794 INFO:teuthology.orchestra.run.smithi195.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-24T23:33:47.856 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-24T23:33:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:34:21.353 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:34:33.501 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-24T23:34:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:34:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-24T23:34:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "repo_digests": [ 2024-04-24T23:34:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-24T23:34:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-04-24T23:34:33.502 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-04-24T23:34:35.805 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-04-24T23:34:35.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-04-24T23:34:35.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-04-24T23:34:35.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-04-24T23:34:35.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-04-24T23:34:35.945 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-04-24T23:34:35.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-04-24T23:34:35.994 INFO:tasks.cephadm:Writing seed config... 2024-04-24T23:34:35.995 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-24T23:34:35.995 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-24T23:34:35.996 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-24T23:34:35.997 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-24T23:34:35.998 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-24T23:34:35.999 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:34:35.999 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-24T23:34:36.016 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 16fa2ae8-0293-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-24T23:34:36.017 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service 2024-04-24T23:34:36.059 DEBUG:teuthology.orchestra.run.smithi107:mgr.x> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x.service 2024-04-24T23:34:36.102 INFO:tasks.cephadm:Bootstrapping... 2024-04-24T23:34:36.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:34:36.293 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:34:36.293 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '16fa2ae8-0293-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.107', '--skip-admin-label'] 2024-04-24T23:34:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-24T23:34:36.294 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying podman|docker is present... 2024-04-24T23:34:36.320 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T23:34:36.320 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying lvm2 is present... 2024-04-24T23:34:36.320 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying time synchronization is in place... 2024-04-24T23:34:36.328 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T23:34:36.328 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T23:34:36.334 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T23:34:36.334 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:34:36.340 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-04-24T23:34:36.346 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-04-24T23:34:36.346 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-04-24T23:34:36.347 INFO:teuthology.orchestra.run.smithi107.stdout:Repeating the final host check... 2024-04-24T23:34:36.380 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T23:34:36.380 INFO:teuthology.orchestra.run.smithi107.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-24T23:34:36.380 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl is present 2024-04-24T23:34:36.380 INFO:teuthology.orchestra.run.smithi107.stdout:lvcreate is present 2024-04-24T23:34:36.386 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T23:34:36.386 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T23:34:36.392 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T23:34:36.392 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:34:36.398 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Host looks OK 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Cluster fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Acquiring lock 139768900140576 on /run/cephadm/16fa2ae8-0293-11ef-bc93-c7b262605968.lock 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Lock 139768900140576 acquired on /run/cephadm/16fa2ae8-0293-11ef-bc93-c7b262605968.lock 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 3300 ... 2024-04-24T23:34:36.405 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 6789 ... 2024-04-24T23:34:36.406 INFO:teuthology.orchestra.run.smithi107.stdout:Base mon IP(s) is [172.21.15.107:3300, 172.21.15.107:6789], mon addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-04-24T23:34:36.409 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-04-24T23:34:36.409 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-04-24T23:34:36.411 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-24T23:34:36.411 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-24T23:34:36.411 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-24T23:34:36.413 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 fe80::6f34:b7ae:a569:134c/64 scope link noprefixroute 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T23:34:36.414 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-04-24T23:34:36.415 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-04-24T23:34:36.415 INFO:teuthology.orchestra.run.smithi107.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-24T23:34:36.415 INFO:teuthology.orchestra.run.smithi107.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-24T23:34:36.416 INFO:teuthology.orchestra.run.smithi107.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-24T23:34:36.910 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-24T23:34:36.910 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-24T23:34:36.910 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Getting image source signatures 2024-04-24T23:34:36.910 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-24T23:34:36.911 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-24T23:34:36.911 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-24T23:34:36.911 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-24T23:34:37.908 INFO:teuthology.orchestra.run.smithi107.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-24T23:34:37.908 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-24T23:34:37.908 INFO:teuthology.orchestra.run.smithi107.stdout:Extracting ceph user uid/gid from container image... 2024-04-24T23:34:38.814 INFO:teuthology.orchestra.run.smithi107.stdout:stat: stdout 167 167 2024-04-24T23:34:38.814 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial keys... 2024-04-24T23:34:39.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQAPlylmTIcpCBAA2e7KdHzOdsn+xtYBey4Nug== 2024-04-24T23:34:40.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQAQlylmkGIYDhAAuWf8wGBHp4PWmfUJqBC3+Q== 2024-04-24T23:34:41.877 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQARlylmfv75ChAAJVYohKGMCryAa2kxnJDzSQ== 2024-04-24T23:34:41.878 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial monmap... 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool for a [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: set fsid to 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:34:42.751 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mon... 2024-04-24T23:34:43.699 INFO:teuthology.orchestra.run.smithi107.stdout:create mon.a on 2024-04-24T23:34:43.910 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-24T23:34:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-24T23:34:44.237 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target. 2024-04-24T23:34:44.237 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target. 2024-04-24T23:34:44.475 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a 2024-04-24T23:34:44.475 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-24T23:34:44.652 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target.wants/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@.service. 2024-04-24T23:34:46.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:46 smithi107 ceph-mon[28744]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T23:34:47.014 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T23:34:47.014 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-04-24T23:34:47.022 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T23:34:47.023 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:34:47.023 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-04-24T23:34:47.023 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T23:34:47.023 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon to start... 2024-04-24T23:34:47.023 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon... 2024-04-24T23:34:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:46 smithi107 bash[28725]: dae77d3ed93de3eb07e6f1d642e141853cbd4e9a1d68d3cb4567557fd414bd13 2024-04-24T23:34:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 systemd[1]: Started Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:34:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T23:34:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: monmap epoch 1 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: last_changed 2024-04-24T23:34:42.177629+0000 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: min_mon_release 19 (squid) 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: election_strategy: 1 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: fsmap 2024-04-24T23:34:47.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T23:34:47.380 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:47 smithi107 ceph-mon[28744]: mgrmap e1: no daemons active 2024-04-24T23:34:50.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:49 smithi107 ceph-mon[28744]: from='client.? 172.21.15.107:0/2469752656' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T23:34:56.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout cluster: 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout id: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout services: 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:56.331 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout data: 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pgs: 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb0235c3700 1 Processor -- start 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb0235c3700 1 -- start start 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb0235c3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb0235c3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c107580 con 0x7fb01c106b90 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb02135f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb02135f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59858/0 (socket says 172.21.15.107:59858) 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.863+0000 7fb02135f700 1 -- 172.21.15.107:0/2256835695 learned_addr learned my addr 172.21.15.107:0/2256835695 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb02135f700 1 -- 172.21.15.107:0/2256835695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c107d90 con 0x7fb01c106b90 2024-04-24T23:34:56.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb02135f700 1 --2- 172.21.15.107:0/2256835695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d095e302fea7825 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb01bfff700 1 -- 172.21.15.107:0/2256835695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb014004160 con 0x7fb01c106b90 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb01bfff700 1 -- 172.21.15.107:0/2256835695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb014004300 con 0x7fb01c106b90 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb01bfff700 1 -- 172.21.15.107:0/2256835695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0140036a0 con 0x7fb01c106b90 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb0235c3700 1 -- 172.21.15.107:0/2256835695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 msgr2=0x7fb01c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.864+0000 7fb0235c3700 1 --2- 172.21.15.107:0/2256835695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).stop 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- 172.21.15.107:0/2256835695 shutdown_connections 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 --2- 172.21.15.107:0/2256835695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c106fb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- 172.21.15.107:0/2256835695 >> 172.21.15.107:0/2256835695 conn(0x7fb01c102170 msgr2=0x7fb01c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- 172.21.15.107:0/2256835695 shutdown_connections 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- 172.21.15.107:0/2256835695 wait complete. 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 Processor -- start 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- start start 2024-04-24T23:34:56.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.865+0000 7fb0235c3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1082c0 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb02135f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb02135f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59860/0 (socket says 172.21.15.107:59860) 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb02135f700 1 -- 172.21.15.107:0/2469752656 learned_addr learned my addr 172.21.15.107:0/2469752656 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb02135f700 1 -- 172.21.15.107:0/2469752656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c19a7d0 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb02135f700 1 --2- 172.21.15.107:0/2469752656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb014000c00 tx=0x7fb014004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb014003db0 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb014011630 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c19aa60 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb014011950 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c19d750 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb014011af0 con 0x7fb01c106b90 2024-04-24T23:34:56.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.866+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb01401c450 con 0x7fb01c106b90 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.867+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb01c063890 con 0x7fb01c106b90 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.869+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb014032020 con 0x7fb01c106b90 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.941+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb01c02dcb0 con 0x7fb01c106b90 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.941+0000 7fb01a7fc700 1 -- 172.21.15.107:0/2469752656 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+310 (secure 0 0 0) 0x7fb014004260 con 0x7fb01c106b90 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 msgr2=0x7fb01c19a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 --2- 172.21.15.107:0/2469752656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb014000c00 tx=0x7fb014004750 comp rx=0 tx=0).stop 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 shutdown_connections 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 --2- 172.21.15.107:0/2469752656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb01c106b90 0x7fb01c19a290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 >> 172.21.15.107:0/2469752656 conn(0x7fb01c102170 msgr2=0x7fb01c190ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 shutdown_connections 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:49.942+0000 7fb0235c3700 1 -- 172.21.15.107:0/2469752656 wait complete. 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:mon is available 2024-04-24T23:34:56.335 INFO:teuthology.orchestra.run.smithi107.stdout:Assimilating anything we can from ceph.conf... 2024-04-24T23:34:57.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:57 smithi107 ceph-mon[28744]: from='client.? 172.21.15.107:0/4136430321' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T23:34:57.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:57 smithi107 ceph-mon[28744]: from='client.? 172.21.15.107:0/4136430321' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T23:34:57.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:57.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T23:34:57.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.978+0000 7f5095a2b700 1 Processor -- start 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.978+0000 7f5095a2b700 1 -- start start 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.978+0000 7f5095a2b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.978+0000 7f5095a2b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f509007c7e0 con 0x7f509007dda0 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.978+0000 7f508effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60830/0 (socket says 172.21.15.107:60830) 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508effd700 1 -- 172.21.15.107:0/1062237558 learned_addr learned my addr 172.21.15.107:0/1062237558 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508effd700 1 -- 172.21.15.107:0/1062237558 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509007c960 con 0x7f509007dda0 2024-04-24T23:34:57.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508effd700 1 --2- 172.21.15.107:0/1062237558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50800098a0 tx=0x7f5080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd2f7643f96be67 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508e7fc700 1 -- 172.21.15.107:0/1062237558 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5080004160 con 0x7f509007dda0 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508e7fc700 1 -- 172.21.15.107:0/1062237558 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080004300 con 0x7f509007dda0 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.979+0000 7f508e7fc700 1 -- 172.21.15.107:0/1062237558 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50800036a0 con 0x7f509007dda0 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- 172.21.15.107:0/1062237558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 msgr2=0x7f509007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 --2- 172.21.15.107:0/1062237558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50800098a0 tx=0x7f5080009bb0 comp rx=0 tx=0).stop 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- 172.21.15.107:0/1062237558 shutdown_connections 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 --2- 172.21.15.107:0/1062237558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509007c2a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- 172.21.15.107:0/1062237558 >> 172.21.15.107:0/1062237558 conn(0x7f50901032c0 msgr2=0x7f50901056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- 172.21.15.107:0/1062237558 shutdown_connections 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- 172.21.15.107:0/1062237558 wait complete. 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 Processor -- start 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.980+0000 7f5095a2b700 1 -- start start 2024-04-24T23:34:57.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5095a2b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5095a2b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f509007ce90 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f508effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f508effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60838/0 (socket says 172.21.15.107:60838) 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f508effd700 1 -- 172.21.15.107:0/4136430321 learned_addr learned my addr 172.21.15.107:0/4136430321 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f508effd700 1 -- 172.21.15.107:0/4136430321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509019dad0 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f508effd700 1 --2- 172.21.15.107:0/4136430321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f5080004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5080003db0 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080011630 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5080011950 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509019dd60 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.981+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901a0a50 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.982+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f508001a460 con 0x7f509007dda0 2024-04-24T23:34:57.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.982+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f508001b780 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.982+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5090063890 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:56.985+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f508001ba20 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.052+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f509002de30 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.055+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f508001bc00 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.056+0000 7f5094a29700 1 -- 172.21.15.107:0/4136430321 <== mon.0 v2:172.21.15.107:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f508001bde0 con 0x7f509007dda0 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 msgr2=0x7f509019d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 --2- 172.21.15.107:0/4136430321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f5080004750 comp rx=0 tx=0).stop 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 shutdown_connections 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 --2- 172.21.15.107:0/4136430321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509007dda0 0x7f509019d590 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 >> 172.21.15.107:0/4136430321 conn(0x7f50901032c0 msgr2=0x7f5090105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 shutdown_connections 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:57.057+0000 7f5095a2b700 1 -- 172.21.15.107:0/4136430321 wait complete. 2024-04-24T23:34:57.634 INFO:teuthology.orchestra.run.smithi107.stdout:Generating new minimal ceph.conf... 2024-04-24T23:34:58.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:58 smithi107 ceph-mon[28744]: from='client.? 172.21.15.107:0/1580812034' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121fdb6700 1 Processor -- start 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121fdb6700 1 -- start start 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121fdb6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121fdb6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f121807bdf0 con 0x7f1218108740 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121db52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121db52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60848/0 (socket says 172.21.15.107:60848) 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121db52700 1 -- 172.21.15.107:0/3382899437 learned_addr learned my addr 172.21.15.107:0/3382899437 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.012+0000 7f121db52700 1 -- 172.21.15.107:0/3382899437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121810b070 con 0x7f1218108740 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121db52700 1 --2- 172.21.15.107:0/3382899437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f12140098a0 tx=0x7f1214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2444de0fbd3932 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:58.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121cb50700 1 -- 172.21.15.107:0/3382899437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214004160 con 0x7f1218108740 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121cb50700 1 -- 172.21.15.107:0/3382899437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1214004300 con 0x7f1218108740 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121cb50700 1 -- 172.21.15.107:0/3382899437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214011b50 con 0x7f1218108740 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 -- 172.21.15.107:0/3382899437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 msgr2=0x7f121810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 --2- 172.21.15.107:0/3382899437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f12140098a0 tx=0x7f1214009bb0 comp rx=0 tx=0).stop 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 -- 172.21.15.107:0/3382899437 shutdown_connections 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 --2- 172.21.15.107:0/3382899437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f121810ab30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 -- 172.21.15.107:0/3382899437 >> 172.21.15.107:0/3382899437 conn(0x7f1218102170 msgr2=0x7f1218104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.013+0000 7f121fdb6700 1 -- 172.21.15.107:0/3382899437 shutdown_connections 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121fdb6700 1 -- 172.21.15.107:0/3382899437 wait complete. 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121fdb6700 1 Processor -- start 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121fdb6700 1 -- start start 2024-04-24T23:34:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121fdb6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121fdb6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f121810c090 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121db52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121db52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60852/0 (socket says 172.21.15.107:60852) 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121db52700 1 -- 172.21.15.107:0/1580812034 learned_addr learned my addr 172.21.15.107:0/1580812034 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.014+0000 7f121db52700 1 -- 172.21.15.107:0/1580812034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1218108500 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f121db52700 1 --2- 172.21.15.107:0/1580812034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1214011b60 tx=0x7f1214011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214003970 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1214003b10 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214024610 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12181066f0 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1218106c30 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1214024ca0 con 0x7f1218108740 2024-04-24T23:34:58.679 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.015+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1214011e00 con 0x7f1218108740 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.016+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1218063890 con 0x7f1218108740 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.018+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f121401a9f0 con 0x7f1218108740 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.085+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1218107160 con 0x7f1218108740 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f120effd700 1 -- 172.21.15.107:0/1580812034 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f1214019aa0 con 0x7f1218108740 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 msgr2=0x7f1218107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f121fdb6700 1 --2- 172.21.15.107:0/1580812034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1214011b60 tx=0x7f1214011c40 comp rx=0 tx=0).stop 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 shutdown_connections 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f121fdb6700 1 --2- 172.21.15.107:0/1580812034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1218108740 0x7f1218107fc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.086+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 >> 172.21.15.107:0/1580812034 conn(0x7f1218102170 msgr2=0x7f1218102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.087+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 shutdown_connections 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:34:58.087+0000 7f121fdb6700 1 -- 172.21.15.107:0/1580812034 wait complete. 2024-04-24T23:34:58.680 INFO:teuthology.orchestra.run.smithi107.stdout:Restarting the monitor... 2024-04-24T23:34:58.942 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:58 smithi107 systemd[1]: Stopping Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:34:59.198 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:58 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a[28739]: 2024-04-24T23:34:58.939+0000 7ff585c40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:34:59.198 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:58 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a[28739]: 2024-04-24T23:34:58.939+0000 7ff585c40700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-24T23:34:59.198 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:58 smithi107 podman[28990]: 2024-04-24 23:34:58.970506997 +0000 UTC m=+0.137780741 container died dae77d3ed93de3eb07e6f1d642e141853cbd4e9a1d68d3cb4567557fd414bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:34:59.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 podman[28990]: 2024-04-24 23:34:59.272889145 +0000 UTC m=+0.440162889 container cleanup dae77d3ed93de3eb07e6f1d642e141853cbd4e9a1d68d3cb4567557fd414bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T23:34:59.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 bash[28990]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a 2024-04-24T23:34:59.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 podman[29004]: 2024-04-24 23:34:59.490389099 +0000 UTC m=+0.517053004 container remove dae77d3ed93de3eb07e6f1d642e141853cbd4e9a1d68d3cb4567557fd414bd13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True) 2024-04-24T23:34:59.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 systemd[1]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T23:34:59.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 systemd[1]: Stopped Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:34:59.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 systemd[1]: Starting Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 podman[29099]: 2024-04-24 23:34:59.816209859 +0000 UTC m=+0.105127588 container create 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:34:59 smithi107 podman[29099]: 2024-04-24 23:34:59.729078967 +0000 UTC m=+0.017996678 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 podman[29099]: 2024-04-24 23:35:00.018552987 +0000 UTC m=+0.307470670 container init 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 podman[29099]: 2024-04-24 23:35:00.022112994 +0000 UTC m=+0.311030680 container start 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: pidfile_write: ignore empty --pid-file 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: load: jerasure load: lrc 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: RocksDB version: 7.9.2 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Git sha 0 2024-04-24T23:35:00.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: DB SUMMARY 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: DB Session ID: EYW326FKRHM0Q8P63LUL 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: CURRENT file: CURRENT 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105161 ; 2024-04-24T23:35:00.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.error_if_exists: 0 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.create_if_missing: 0 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.paranoid_checks: 1 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.env: 0x5573ca746c60 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.info_log: 0x5573cb0dfaa0 2024-04-24T23:35:00.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.statistics: (nil) 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.use_fsync: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_log_file_size: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_fallocate: 1 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.use_direct_reads: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.db_log_dir: 2024-04-24T23:35:00.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.wal_dir: 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.write_buffer_manager: 0x5573cb0b21e0 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T23:35:00.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.unordered_write: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.row_cache: None 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.wal_filter: None 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.two_write_queues: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.wal_compression: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.atomic_flush: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.log_readahead_size: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T23:35:00.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_background_jobs: 2 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_background_compactions: -1 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_subcompactions: 1 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T23:35:00.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_open_files: -1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_background_flushes: -1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Compression algorithms supported: 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kZSTD supported: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kXpressCompression supported: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kZlibCompression supported: 1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kSnappyCompression supported: 1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kLZ4Compression supported: 1 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: kBZip2Compression supported: 0 2024-04-24T23:35:00.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.merge_operator: 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_filter: None 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5573cb0dfba0) 2024-04-24T23:35:00.086 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 4 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x5573cb0e3090 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-04-24T23:35:00.087 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-04-24T23:35:00.088 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: bloomfilter 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-04-24T23:35:00.089 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 5 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_auto_readahead_size: 262144 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: prepopulate_block_cache: 0 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: initial_auto_readahead_size: 8192 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression: NoCompression 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T23:35:00.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.num_levels: 7 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T23:35:00.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T23:35:00.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T23:35:00.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.table_properties_collectors: 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.inplace_update_support: 0 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T23:35:00.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.bloom_locality: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.max_successive_merges: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.ttl: 2592000 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enable_blob_files: false 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.min_blob_size: 0 2024-04-24T23:35:00.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ff04057a-b976-407d-a229-7bd38187463e 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001700051959, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-24T23:35:00.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001700052871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 98034, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14217, "raw_average_key_size": 49, "raw_value_size": 90305, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714001700, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff04057a-b976-407d-a229-7bd38187463e", "db_session_id": "EYW326FKRHM0Q8P63LUL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001700052960, "job": 1, "event": "recovery_finished"} 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5573cb1d4000 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: DB pointer 0x5573cb1be000 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T23:35:00.097 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 99.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 99.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 169.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:35:00.098 INFO:journalctl@ceph.mon.a.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T23:35:00.099 INFO:teuthology.orchestra.run.smithi107.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 169.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T23:35:00.099 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 33.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 33.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache BinnedLRUCache@0x5573cb0e3090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???) e1 preinit fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).mds e1 new map 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).mds e1 print_map 2024-04-24T23:35:00.100 INFO:journalctl@ceph.mon.a.smithi107.stdout: e1 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: -1 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout: No filesystems configured 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 bash[29099]: 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 2024-04-24T23:35:00.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 systemd[1]: Started Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:35:00.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T23:35:00.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: monmap epoch 1 2024-04-24T23:35:00.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: last_changed 2024-04-24T23:34:42.177629+0000 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: min_mon_release 19 (squid) 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: election_strategy: 1 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: fsmap 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T23:35:00.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:00 smithi107 ceph-mon[29118]: mgrmap e1: no daemons active 2024-04-24T23:35:01.169 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.514+0000 7fa5a8dc9700 1 Processor -- start 2024-04-24T23:35:01.173 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.514+0000 7fa5a8dc9700 1 -- start start 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.515+0000 7fa5a8dc9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.515+0000 7fa5a8dc9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a4108690 con 0x7fa5a4107ca0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.515+0000 7fa5a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.515+0000 7fa5a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60862/0 (socket says 172.21.15.107:60862) 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.515+0000 7fa5a259c700 1 -- 172.21.15.107:0/3310534430 learned_addr learned my addr 172.21.15.107:0/3310534430 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a259c700 1 -- 172.21.15.107:0/3310534430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a4108e50 con 0x7fa5a4107ca0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a259c700 1 --2- 172.21.15.107:0/3310534430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5980098a0 tx=0x7fa598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a298b6a3259ce817 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a159a700 1 -- 172.21.15.107:0/3310534430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa598004160 con 0x7fa5a4107ca0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a159a700 1 -- 172.21.15.107:0/3310534430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa598004300 con 0x7fa5a4107ca0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a159a700 1 -- 172.21.15.107:0/3310534430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa598011b50 con 0x7fa5a4107ca0 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/3310534430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 msgr2=0x7fa5a41080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.516+0000 7fa5a8dc9700 1 --2- 172.21.15.107:0/3310534430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5980098a0 tx=0x7fa598009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/3310534430 shutdown_connections 2024-04-24T23:35:01.174 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 --2- 172.21.15.107:0/3310534430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a41080c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/3310534430 >> 172.21.15.107:0/3310534430 conn(0x7fa5a41032a0 msgr2=0x7fa5a41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/3310534430 shutdown_connections 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/3310534430 wait complete. 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 Processor -- start 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.517+0000 7fa5a8dc9700 1 -- start start 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a8dc9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a8dc9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a410c200 con 0x7fa5a4107ca0 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60866/0 (socket says 172.21.15.107:60866) 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a259c700 1 -- 172.21.15.107:0/2756984264 learned_addr learned my addr 172.21.15.107:0/2756984264 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a259c700 1 -- 172.21.15.107:0/2756984264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a419e3f0 con 0x7fa5a4107ca0 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a259c700 1 --2- 172.21.15.107:0/2756984264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa598011850 tx=0x7fa598011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa598003af0 con 0x7fa5a4107ca0 2024-04-24T23:35:01.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa598020070 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a419e680 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5980195a0 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.518+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a41a1370 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.519+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa598019c30 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.519+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa598011e00 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.519+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a4063890 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.522+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa598003c90 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.587+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa5a41a1540 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.591+0000 7fa5937fe700 1 -- 172.21.15.107:0/2756984264 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa59801a430 con 0x7fa5a4107ca0 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 msgr2=0x7fa5a419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 --2- 172.21.15.107:0/2756984264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa598011850 tx=0x7fa598011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 shutdown_connections 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 --2- 172.21.15.107:0/2756984264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5a4107ca0 0x7fa5a419deb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:01.176 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 >> 172.21.15.107:0/2756984264 conn(0x7fa5a41032a0 msgr2=0x7fa5a4103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:01.177 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 shutdown_connections 2024-04-24T23:35:01.177 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:00.592+0000 7fa5a8dc9700 1 -- 172.21.15.107:0/2756984264 wait complete. 2024-04-24T23:35:01.177 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-24T23:35:01.199 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:35:01.199 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mgr... 2024-04-24T23:35:01.200 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-24T23:35:01.200 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-24T23:35:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x 2024-04-24T23:35:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-24T23:35:01.585 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968.target.wants/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@.service. 2024-04-24T23:35:01.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2756984264' entity='client.admin' 2024-04-24T23:35:01.828 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:01 smithi107 systemd[1]: Starting Ceph mgr.x for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:35:02.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:01 smithi107 podman[29337]: 2024-04-24 23:35:01.828033734 +0000 UTC m=+0.108801393 container create b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-24T23:35:02.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:01 smithi107 podman[29337]: 2024-04-24 23:35:01.736852257 +0000 UTC m=+0.017619928 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:35:02.236 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T23:35:02.236 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-04-24T23:35:02.243 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T23:35:02.243 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:35:02.243 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-04-24T23:35:02.244 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T23:35:02.252 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T23:35:02.252 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr to start... 2024-04-24T23:35:02.259 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr... 2024-04-24T23:35:02.439 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 podman[29337]: 2024-04-24 23:35:02.147003594 +0000 UTC m=+0.427771253 container init b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-24T23:35:02.439 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 podman[29337]: 2024-04-24 23:35:02.149382468 +0000 UTC m=+0.430150131 container start b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-24T23:35:02.439 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 bash[29337]: b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf 2024-04-24T23:35:02.439 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 systemd[1]: Started Ceph mgr.x for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:35:02.440 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:02.307+0000 7f8821813200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:35:02.708 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:02.491+0000 7f8821813200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:02.708 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:02.598+0000 7f8821813200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:35:03.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:02 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/864413890' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T23:35:03.010 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:02.924+0000 7f8821813200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:35:03.010 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:03 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:03.009+0000 7f8821813200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.427 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T23:35:03.428 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T23:35:03.428 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-04-24T23:35:03.428 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:03.433 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T23:35:03.433 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T23:35:03.433 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T23:35:03.434 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T23:35:03.435 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.436 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T23:34:45.996851+0000", 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.725+0000 7f6a7d5a6700 1 Processor -- start 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.725+0000 7f6a7d5a6700 1 -- start start 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.725+0000 7f6a7d5a6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.725+0000 7f6a7d5a6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78107580 con 0x7f6a78106b90 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60884/0 (socket says 172.21.15.107:60884) 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a76ffd700 1 -- 172.21.15.107:0/4102711390 learned_addr learned my addr 172.21.15.107:0/4102711390 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:03.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a76ffd700 1 -- 172.21.15.107:0/4102711390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a78107d90 con 0x7f6a78106b90 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a76ffd700 1 --2- 172.21.15.107:0/4102711390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0098a0 tx=0x7f6a6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13595f095377aabf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a75ffb700 1 -- 172.21.15.107:0/4102711390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a6c004160 con 0x7f6a78106b90 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a75ffb700 1 -- 172.21.15.107:0/4102711390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a6c004300 con 0x7f6a78106b90 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.726+0000 7f6a75ffb700 1 -- 172.21.15.107:0/4102711390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a6c011b50 con 0x7f6a78106b90 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/4102711390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 msgr2=0x7f6a78106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 --2- 172.21.15.107:0/4102711390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0098a0 tx=0x7f6a6c009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/4102711390 shutdown_connections 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 --2- 172.21.15.107:0/4102711390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a78106fb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/4102711390 >> 172.21.15.107:0/4102711390 conn(0x7f6a78102130 msgr2=0x7f6a78104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/4102711390 shutdown_connections 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.727+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/4102711390 wait complete. 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a7d5a6700 1 Processor -- start 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a7d5a6700 1 -- start start 2024-04-24T23:35:03.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a7d5a6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a7d5a6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7810b140 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a76ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60900/0 (socket says 172.21.15.107:60900) 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a76ffd700 1 -- 172.21.15.107:0/864413890 learned_addr learned my addr 172.21.15.107:0/864413890 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.728+0000 7f6a76ffd700 1 -- 172.21.15.107:0/864413890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7819c1f0 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a76ffd700 1 --2- 172.21.15.107:0/864413890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009fb0 tx=0x7f6a6c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a6c0038b0 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7819c480 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a6c011e50 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a6c019550 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7819f170 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a6c019be0 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.729+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a6c024730 con 0x7f6a78106b90 2024-04-24T23:35:03.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.730+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a78063890 con 0x7f6a78106b90 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.732+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6a6c003a50 con 0x7f6a78106b90 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.804+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6a7802dd80 con 0x7f6a78106b90 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.805+0000 7f6a5ffff700 1 -- 172.21.15.107:0/864413890 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6a6c0249d0 con 0x7f6a78106b90 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 msgr2=0x7f6a7819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 --2- 172.21.15.107:0/864413890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009fb0 tx=0x7f6a6c011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 shutdown_connections 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 --2- 172.21.15.107:0/864413890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a78106b90 0x7f6a7819bcb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 >> 172.21.15.107:0/864413890 conn(0x7f6a78102130 msgr2=0x7f6a78103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 shutdown_connections 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:02.806+0000 7f6a7d5a6700 1 -- 172.21.15.107:0/864413890 wait complete. 2024-04-24T23:35:03.440 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (1/15)... 2024-04-24T23:35:04.540 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:04.201+0000 7f8821813200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:35:04.540 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:04.371+0000 7f8821813200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:04.540 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:04.540+0000 7f8821813200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:35:05.118 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:04.824+0000 7f8821813200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:35:05.118 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:04.916+0000 7f8821813200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:35:05.118 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:05.117+0000 7f8821813200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:35:05.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:05.308+0000 7f8821813200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:06.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3821106342' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T23:35:06.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:05.924+0000 7f8821813200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:35:06.489 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-04-24T23:35:06.490 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T23:35:06.491 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.492 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:06.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T23:34:45.996851+0000", 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.841+0000 7ff14e93d700 1 Processor -- start 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.841+0000 7ff14e93d700 1 -- start start 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff14e93d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff14e93d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff148108c00 con 0x7ff148108210 2024-04-24T23:35:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff147fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff147fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60902/0 (socket says 172.21.15.107:60902) 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff147fff700 1 -- 172.21.15.107:0/3044841171 learned_addr learned my addr 172.21.15.107:0/3044841171 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff147fff700 1 -- 172.21.15.107:0/3044841171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff148105d10 con 0x7ff148108210 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff147fff700 1 --2- 172.21.15.107:0/3044841171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1380098a0 tx=0x7ff138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1395e653de31e606 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.842+0000 7ff146ffd700 1 -- 172.21.15.107:0/3044841171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff138004160 con 0x7ff148108210 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff146ffd700 1 -- 172.21.15.107:0/3044841171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff138004300 con 0x7ff148108210 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff146ffd700 1 -- 172.21.15.107:0/3044841171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff138011b50 con 0x7ff148108210 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 -- 172.21.15.107:0/3044841171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 msgr2=0x7ff148108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 --2- 172.21.15.107:0/3044841171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1380098a0 tx=0x7ff138009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 -- 172.21.15.107:0/3044841171 shutdown_connections 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 --2- 172.21.15.107:0/3044841171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff148108630 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 -- 172.21.15.107:0/3044841171 >> 172.21.15.107:0/3044841171 conn(0x7ff1481018f0 msgr2=0x7ff148103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 -- 172.21.15.107:0/3044841171 shutdown_connections 2024-04-24T23:35:06.495 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.843+0000 7ff14e93d700 1 -- 172.21.15.107:0/3044841171 wait complete. 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff14e93d700 1 Processor -- start 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff14e93d700 1 -- start start 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff14e93d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff14e93d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff148106d90 con 0x7ff148108210 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff147fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff147fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60912/0 (socket says 172.21.15.107:60912) 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.844+0000 7ff147fff700 1 -- 172.21.15.107:0/3821106342 learned_addr learned my addr 172.21.15.107:0/3821106342 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff147fff700 1 -- 172.21.15.107:0/3821106342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14819e320 con 0x7ff148108210 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff147fff700 1 --2- 172.21.15.107:0/3821106342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff138011b60 tx=0x7ff138011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff138003970 con 0x7ff148108210 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff138003b10 con 0x7ff148108210 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14819e5b0 con 0x7ff148108210 2024-04-24T23:35:06.496 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff138024610 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1481a12a0 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff138024ca0 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.845+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff138011e00 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.846+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff148063890 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.848+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff13801a9f0 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.921+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff1481973e0 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.922+0000 7ff1457fa700 1 -- 172.21.15.107:0/3821106342 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff1481973e0 con 0x7ff148108210 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 msgr2=0x7ff14819dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 --2- 172.21.15.107:0/3821106342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff138011b60 tx=0x7ff138011c40 comp rx=0 tx=0).stop 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 shutdown_connections 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 --2- 172.21.15.107:0/3821106342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff148108210 0x7ff14819dde0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 >> 172.21.15.107:0/3821106342 conn(0x7ff1481018f0 msgr2=0x7ff1481944c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 shutdown_connections 2024-04-24T23:35:06.497 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:05.923+0000 7ff14e93d700 1 -- 172.21.15.107:0/3821106342 wait complete. 2024-04-24T23:35:06.498 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (2/15)... 2024-04-24T23:35:06.628 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:06.354+0000 7f8821813200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:35:06.628 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:06.451+0000 7f8821813200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:35:07.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:06.654+0000 7f8821813200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:35:07.529 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.193+0000 7f8821813200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:35:07.529 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.286+0000 7f8821813200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:35:07.868 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.529+0000 7f8821813200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:35:07.868 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.660+0000 7f8821813200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:35:07.868 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.755+0000 7f8821813200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:35:07.868 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.868+0000 7f8821813200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:08.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:07.957+0000 7f8821813200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:35:08.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:08.771+0000 7f8821813200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:08.994+0000 7f8821813200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2699762435' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: Activating manager daemon x 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: mgrmap e2: x(active, starting, since 0.00356761s) 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T23:35:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: Manager daemon x is now available 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T23:35:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:09 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T23:35:09.595 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:09.595 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:09.595 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-24T23:35:09.596 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T23:35:09.597 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T23:35:09.605 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T23:35:09.605 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.606 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:09.607 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T23:34:45.996851+0000", 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.893+0000 7f016781f700 1 Processor -- start 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f016781f700 1 -- start start 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f016781f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f016781f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0160108670 con 0x7f0160107c80 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f01655bb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:09.608 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f01655bb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38162/0 (socket says 172.21.15.107:38162) 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f01655bb700 1 -- 172.21.15.107:0/4262014205 learned_addr learned my addr 172.21.15.107:0/4262014205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f01655bb700 1 -- 172.21.15.107:0/4262014205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0160108e30 con 0x7f0160107c80 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f01655bb700 1 --2- 172.21.15.107:0/4262014205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f015c0098a0 tx=0x7f015c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9946fd93a24ab34 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.894+0000 7f0157fff700 1 -- 172.21.15.107:0/4262014205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f015c004160 con 0x7f0160107c80 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f0157fff700 1 -- 172.21.15.107:0/4262014205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f015c004300 con 0x7f0160107c80 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f0157fff700 1 -- 172.21.15.107:0/4262014205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f015c011b50 con 0x7f0160107c80 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 -- 172.21.15.107:0/4262014205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 msgr2=0x7f01601080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 --2- 172.21.15.107:0/4262014205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f015c0098a0 tx=0x7f015c009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 -- 172.21.15.107:0/4262014205 shutdown_connections 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 --2- 172.21.15.107:0/4262014205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f01601080a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 -- 172.21.15.107:0/4262014205 >> 172.21.15.107:0/4262014205 conn(0x7f0160103260 msgr2=0x7f0160105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 -- 172.21.15.107:0/4262014205 shutdown_connections 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.895+0000 7f016781f700 1 -- 172.21.15.107:0/4262014205 wait complete. 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f016781f700 1 Processor -- start 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f016781f700 1 -- start start 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f016781f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f016781f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f016010c1e0 con 0x7f0160107c80 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f01655bb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f01655bb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38176/0 (socket says 172.21.15.107:38176) 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f01655bb700 1 -- 172.21.15.107:0/2699762435 learned_addr learned my addr 172.21.15.107:0/2699762435 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f01655bb700 1 -- 172.21.15.107:0/2699762435 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016019c160 con 0x7f0160107c80 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.896+0000 7f01655bb700 1 --2- 172.21.15.107:0/2699762435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f015c011670 tx=0x7f015c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:09.610 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f015c0039c0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f015c011e50 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016019c3f0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f015c01a5c0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016019f0e0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f015c01ac00 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.897+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f015c019700 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.898+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0160063890 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.900+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f015c003b60 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.971+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f016019f2b0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.972+0000 7f01567fc700 1 -- 172.21.15.107:0/2699762435 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f015c0199a0 con 0x7f0160107c80 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 msgr2=0x7f016019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 --2- 172.21.15.107:0/2699762435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f015c011670 tx=0x7f015c011b60 comp rx=0 tx=0).stop 2024-04-24T23:35:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 shutdown_connections 2024-04-24T23:35:09.612 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 --2- 172.21.15.107:0/2699762435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0160107c80 0x7f016019bc20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:09.612 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 >> 172.21.15.107:0/2699762435 conn(0x7f0160103260 msgr2=0x7f0160103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:09.612 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 shutdown_connections 2024-04-24T23:35:09.612 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:08.973+0000 7f016781f700 1 -- 172.21.15.107:0/2699762435 wait complete. 2024-04-24T23:35:09.612 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (3/15)... 2024-04-24T23:35:10.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:10 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' 2024-04-24T23:35:10.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:10 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' 2024-04-24T23:35:10.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:10 smithi107 ceph-mon[29118]: from='mgr.14100 172.21.15.107:0/3664016868' entity='mgr.x' 2024-04-24T23:35:10.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:10 smithi107 ceph-mon[29118]: mgrmap e3: x(active, since 1.00712s) 2024-04-24T23:35:12.312 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:12 smithi107 ceph-mon[29118]: mgrmap e4: x(active, since 2s) 2024-04-24T23:35:12.829 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T23:35:12.830 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T23:35:12.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T23:35:12.840 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T23:35:12.841 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T23:34:45.996851+0000", 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:12.842 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.027+0000 7f0f0997c700 1 Processor -- start 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.027+0000 7f0f0997c700 1 -- start start 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.027+0000 7f0f0997c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.027+0000 7f0f0997c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0407c850 con 0x7f0f0407ddd0 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f02ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f02ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38224/0 (socket says 172.21.15.107:38224) 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f02ffd700 1 -- 172.21.15.107:0/520614083 learned_addr learned my addr 172.21.15.107:0/520614083 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f02ffd700 1 -- 172.21.15.107:0/520614083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0407c9d0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f02ffd700 1 --2- 172.21.15.107:0/520614083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80098a0 tx=0x7f0ef8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55363a556187fb38 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f01ffb700 1 -- 172.21.15.107:0/520614083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ef8004160 con 0x7f0f0407ddd0 2024-04-24T23:35:12.843 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f01ffb700 1 -- 172.21.15.107:0/520614083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0ef8004300 con 0x7f0f0407ddd0 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.028+0000 7f0f01ffb700 1 -- 172.21.15.107:0/520614083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ef8011b50 con 0x7f0f0407ddd0 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 -- 172.21.15.107:0/520614083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 msgr2=0x7f0f0407c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 --2- 172.21.15.107:0/520614083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80098a0 tx=0x7f0ef8009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 -- 172.21.15.107:0/520614083 shutdown_connections 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 --2- 172.21.15.107:0/520614083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0407c280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 -- 172.21.15.107:0/520614083 >> 172.21.15.107:0/520614083 conn(0x7f0f041032c0 msgr2=0x7f0f041056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 -- 172.21.15.107:0/520614083 shutdown_connections 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 -- 172.21.15.107:0/520614083 wait complete. 2024-04-24T23:35:12.844 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.029+0000 7f0f0997c700 1 Processor -- start 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0997c700 1 -- start start 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0997c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0997c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410e250 con 0x7f0f0407ddd0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f02ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f02ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38226/0 (socket says 172.21.15.107:38226) 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f02ffd700 1 -- 172.21.15.107:0/3634069109 learned_addr learned my addr 172.21.15.107:0/3634069109 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f02ffd700 1 -- 172.21.15.107:0/3634069109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0419e330 con 0x7f0f0407ddd0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f02ffd700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8011670 tx=0x7f0ef8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ef80039c0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0419e5c0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0ef8011e50 con 0x7f0f0407ddd0 2024-04-24T23:35:12.845 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ef801a5c0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.030+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f041a12b0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.031+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f0ef801a870 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.031+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f04063890 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.031+0000 7f0f0897a700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 0x7f0eec03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.031+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0ef804d3e0 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.032+0000 7f0f027fc700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 0x7f0eec03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.032+0000 7f0f027fc700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 0x7f0eec03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4006fd0 tx=0x7f0ef4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.037+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ef8019c70 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.312+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0f041a1480 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.313+0000 7f0f0897a700 1 -- 172.21.15.107:0/3634069109 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0f041a1480 con 0x7f0f0407ddd0 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 msgr2=0x7f0eec03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 0x7f0eec03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4006fd0 tx=0x7f0ef4006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 msgr2=0x7f0f0419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:12.846 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8011670 tx=0x7f0ef8011b60 comp rx=0 tx=0).stop 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 shutdown_connections 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f0eec039000 0x7f0eec03b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 --2- 172.21.15.107:0/3634069109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f0407ddd0 0x7f0f0419ddf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 >> 172.21.15.107:0/3634069109 conn(0x7f0f041032c0 msgr2=0x7f0f04103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 shutdown_connections 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:12.316+0000 7f0f0997c700 1 -- 172.21.15.107:0/3634069109 wait complete. 2024-04-24T23:35:12.847 INFO:teuthology.orchestra.run.smithi107.stdout:mgr is available 2024-04-24T23:35:13.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:13 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3634069109' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T23:35:13.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:13.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T23:35:13.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:13.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T23:35:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.223+0000 7f7a39163700 1 Processor -- start 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.223+0000 7f7a39163700 1 -- start start 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.223+0000 7f7a39163700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.223+0000 7f7a39163700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7a34107580 con 0x7f7a34106b90 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.223+0000 7f7a32d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a32d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38238/0 (socket says 172.21.15.107:38238) 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a32d9d700 1 -- 172.21.15.107:0/3500827057 learned_addr learned my addr 172.21.15.107:0/3500827057 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a32d9d700 1 -- 172.21.15.107:0/3500827057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a34107d90 con 0x7f7a34106b90 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a32d9d700 1 --2- 172.21.15.107:0/3500827057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7a240098a0 tx=0x7f7a24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a777b454ad1143 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a31d9b700 1 -- 172.21.15.107:0/3500827057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a24004160 con 0x7f7a34106b90 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a31d9b700 1 -- 172.21.15.107:0/3500827057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a24004300 con 0x7f7a34106b90 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.224+0000 7f7a31d9b700 1 -- 172.21.15.107:0/3500827057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a24011b50 con 0x7f7a34106b90 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 -- 172.21.15.107:0/3500827057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 msgr2=0x7f7a34106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:13.957 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 --2- 172.21.15.107:0/3500827057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7a240098a0 tx=0x7f7a24009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 -- 172.21.15.107:0/3500827057 shutdown_connections 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 --2- 172.21.15.107:0/3500827057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a34106fb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 -- 172.21.15.107:0/3500827057 >> 172.21.15.107:0/3500827057 conn(0x7f7a34102110 msgr2=0x7f7a34104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 -- 172.21.15.107:0/3500827057 shutdown_connections 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 -- 172.21.15.107:0/3500827057 wait complete. 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.225+0000 7f7a39163700 1 Processor -- start 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a39163700 1 -- start start 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a39163700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a39163700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3410b140 con 0x7f7a34106b90 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a32d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a32d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38254/0 (socket says 172.21.15.107:38254) 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a32d9d700 1 -- 172.21.15.107:0/2333797161 learned_addr learned my addr 172.21.15.107:0/2333797161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:13.958 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a32d9d700 1 -- 172.21.15.107:0/2333797161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3419c240 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a32d9d700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a24011850 tx=0x7f7a24011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a24003af0 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a24020070 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3419c4d0 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a240195a0 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.226+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3419f1b0 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.227+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f7a2401a430 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.227+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a34063890 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.228+0000 7f7a1bfff700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 0x7f7a1c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.228+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7a24011e00 con 0x7f7a34106b90 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.228+0000 7f7a3259c700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 0x7f7a1c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.228+0000 7f7a3259c700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 0x7f7a1c03b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:13.959 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.232+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a2400e260 con 0x7f7a34106b90 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.426+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7a340008d0 con 0x7f7a34106b90 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.427+0000 7f7a1bfff700 1 -- 172.21.15.107:0/2333797161 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f7a2404ce40 con 0x7f7a34106b90 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 msgr2=0x7f7a1c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 0x7f7a1c03b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 msgr2=0x7f7a3419bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a24011850 tx=0x7f7a24011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 shutdown_connections 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f7a1c039000 0x7f7a1c03b4c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 --2- 172.21.15.107:0/2333797161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a34106b90 0x7f7a3419bd00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 >> 172.21.15.107:0/2333797161 conn(0x7f7a34102110 msgr2=0x7f7a34103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 shutdown_connections 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:13.429+0000 7f7a39163700 1 -- 172.21.15.107:0/2333797161 wait complete. 2024-04-24T23:35:13.960 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling cephadm module... 2024-04-24T23:35:14.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2333797161' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T23:35:15.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2602632584' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T23:35:15.359 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: ignoring --setuser ceph since I am not root 2024-04-24T23:35:15.359 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: ignoring --setgroup ceph since I am not root 2024-04-24T23:35:15.359 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:15.212+0000 7f7ce4838200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:35:15.618 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:15.397+0000 7f7ce4838200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:15.618 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:15.499+0000 7f7ce4838200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:35:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15eccd700 1 Processor -- start 2024-04-24T23:35:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15eccd700 1 -- start start 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15eccd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15eccd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb158106980 con 0x7fb158105f90 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15ca69700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15ca69700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38264/0 (socket says 172.21.15.107:38264) 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.387+0000 7fb15ca69700 1 -- 172.21.15.107:0/1522658823 learned_addr learned my addr 172.21.15.107:0/1522658823 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb15ca69700 1 -- 172.21.15.107:0/1522658823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb158108f20 con 0x7fb158105f90 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb15ca69700 1 --2- 172.21.15.107:0/1522658823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0098a0 tx=0x7fb14c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66701d297b565bd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb1577fe700 1 -- 172.21.15.107:0/1522658823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb14c004160 con 0x7fb158105f90 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb1577fe700 1 -- 172.21.15.107:0/1522658823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb14c004300 con 0x7fb158105f90 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb1577fe700 1 -- 172.21.15.107:0/1522658823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb14c011b50 con 0x7fb158105f90 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb15eccd700 1 -- 172.21.15.107:0/1522658823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 msgr2=0x7fb1581063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.388+0000 7fb15eccd700 1 --2- 172.21.15.107:0/1522658823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0098a0 tx=0x7fb14c009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- 172.21.15.107:0/1522658823 shutdown_connections 2024-04-24T23:35:15.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 --2- 172.21.15.107:0/1522658823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb1581063b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- 172.21.15.107:0/1522658823 >> 172.21.15.107:0/1522658823 conn(0x7fb158101940 msgr2=0x7fb158103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- 172.21.15.107:0/1522658823 shutdown_connections 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- 172.21.15.107:0/1522658823 wait complete. 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 Processor -- start 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- start start 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.389+0000 7fb15eccd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb15810c2d0 con 0x7fb158105f90 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15ca69700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15ca69700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38280/0 (socket says 172.21.15.107:38280) 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15ca69700 1 -- 172.21.15.107:0/2602632584 learned_addr learned my addr 172.21.15.107:0/2602632584 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15ca69700 1 -- 172.21.15.107:0/2602632584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15819c210 con 0x7fb158105f90 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15ca69700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009fb0 tx=0x7fb14c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb14c0038b0 con 0x7fb158105f90 2024-04-24T23:35:15.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb14c011e50 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15819c4a0 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb14c019550 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.390+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb15819f180 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.391+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fb14c01a430 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.391+0000 7fb155ffb700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 0x7fb14803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.391+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb158063890 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.391+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb14c04d1a0 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.391+0000 7fb157fff700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 0x7fb14803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.392+0000 7fb157fff700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 0x7fb14803b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb144006fd0 tx=0x7fb144006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.396+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb14c016db0 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:14.644+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fb15819f3f0 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.041+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fb14c019850 con 0x7fb158105f90 2024-04-24T23:35:15.629 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.043+0000 7fb155ffb700 1 -- 172.21.15.107:0/2602632584 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fb14c01a720 con 0x7fb158105f90 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 msgr2=0x7fb14803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 0x7fb14803b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb144006fd0 tx=0x7fb144006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 msgr2=0x7fb15819bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009fb0 tx=0x7fb14c011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 shutdown_connections 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb148038fb0 0x7fb14803b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 --2- 172.21.15.107:0/2602632584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb158105f90 0x7fb15819bcd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 >> 172.21.15.107:0/2602632584 conn(0x7fb158101940 msgr2=0x7fb1581024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 shutdown_connections 2024-04-24T23:35:15.630 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:15.044+0000 7fb15eccd700 1 -- 172.21.15.107:0/2602632584 wait complete. 2024-04-24T23:35:15.869 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:15.812+0000 7f7ce4838200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:35:16.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2602632584' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T23:35:16.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:16 smithi107 ceph-mon[29118]: mgrmap e5: x(active, since 6s) 2024-04-24T23:35:16.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:15.900+0000 7f7ce4838200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.053+0000 7fb2be94a700 1 Processor -- start 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.053+0000 7fb2be94a700 1 -- start start 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.053+0000 7fb2be94a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.053+0000 7fb2be94a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b807be30 con 0x7fb2b8108710 2024-04-24T23:35:16.934 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38292/0 (socket says 172.21.15.107:38292) 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b7fff700 1 -- 172.21.15.107:0/639301322 learned_addr learned my addr 172.21.15.107:0/639301322 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b7fff700 1 -- 172.21.15.107:0/639301322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b810b040 con 0x7fb2b8108710 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b7fff700 1 --2- 172.21.15.107:0/639301322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0098a0 tx=0x7fb2ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3647ff3508689f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b6ffd700 1 -- 172.21.15.107:0/639301322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac004160 con 0x7fb2b8108710 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b6ffd700 1 -- 172.21.15.107:0/639301322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2ac004300 con 0x7fb2b8108710 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.054+0000 7fb2b6ffd700 1 -- 172.21.15.107:0/639301322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac011b50 con 0x7fb2b8108710 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 -- 172.21.15.107:0/639301322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 msgr2=0x7fb2b810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 --2- 172.21.15.107:0/639301322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0098a0 tx=0x7fb2ac009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 -- 172.21.15.107:0/639301322 shutdown_connections 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 --2- 172.21.15.107:0/639301322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b810ab00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 -- 172.21.15.107:0/639301322 >> 172.21.15.107:0/639301322 conn(0x7fb2b8102160 msgr2=0x7fb2b8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:16.935 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 -- 172.21.15.107:0/639301322 shutdown_connections 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.055+0000 7fb2be94a700 1 -- 172.21.15.107:0/639301322 wait complete. 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2be94a700 1 Processor -- start 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2be94a700 1 -- start start 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2be94a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2be94a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b810c070 con 0x7fb2b8108710 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2b7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2b7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38302/0 (socket says 172.21.15.107:38302) 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.056+0000 7fb2b7fff700 1 -- 172.21.15.107:0/1142571812 learned_addr learned my addr 172.21.15.107:0/1142571812 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2b7fff700 1 -- 172.21.15.107:0/1142571812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b81084d0 con 0x7fb2b8108710 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2b7fff700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009fb0 tx=0x7fb2ac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac0038b0 con 0x7fb2b8108710 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2ac011e50 con 0x7fb2b8108710 2024-04-24T23:35:16.936 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b81066c0 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac019420 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.057+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b8106c00 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fb2ac019720 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2b57fa700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 0x7fb2a003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb2ac04e2c0 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2b77fe700 1 -- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 msgr2=0x7fb2a003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2b77fe700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 0x7fb2a003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.058+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b8063890 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.063+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb2ac023c90 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.259+0000 7fb2b77fe700 1 -- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 msgr2=0x7fb2a003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.259+0000 7fb2b77fe700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 0x7fb2a003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.356+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb2b8107410 con 0x7fb2b8108710 2024-04-24T23:35:16.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.357+0000 7fb2b57fa700 1 -- 172.21.15.107:0/1142571812 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fb2ac00bc70 con 0x7fb2b8108710 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 msgr2=0x7fb2a003b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 0x7fb2a003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 msgr2=0x7fb2b8107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009fb0 tx=0x7fb2ac011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 shutdown_connections 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7fb2a0039000 0x7fb2a003b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 --2- 172.21.15.107:0/1142571812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b8108710 0x7fb2b8107f90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 >> 172.21.15.107:0/1142571812 conn(0x7fb2b8102160 msgr2=0x7fb2b8102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 shutdown_connections 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:16.359+0000 7fb2be94a700 1 -- 172.21.15.107:0/1142571812 wait complete. 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-04-24T23:35:16.938 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 5... 2024-04-24T23:35:17.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1142571812' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T23:35:17.328 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:17.111+0000 7f7ce4838200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:35:17.328 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:17.288+0000 7f7ce4838200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:17.628 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:17.461+0000 7f7ce4838200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:35:18.042 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:17.749+0000 7f7ce4838200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:35:18.042 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:17.841+0000 7f7ce4838200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:35:18.043 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:18 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:18.042+0000 7f7ce4838200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:35:18.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:18 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:18.234+0000 7f7ce4838200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:19.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:18 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:18.828+0000 7f7ce4838200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:35:19.547 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:19 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:19.250+0000 7f7ce4838200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:35:19.547 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:19 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:19.347+0000 7f7ce4838200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:35:19.547 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:19 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:19.547+0000 7f7ce4838200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:35:20.348 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.085+0000 7f7ce4838200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:35:20.348 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.177+0000 7f7ce4838200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:35:20.614 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.422+0000 7f7ce4838200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:35:20.614 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.560+0000 7f7ce4838200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:35:20.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.659+0000 7f7ce4838200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:35:20.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.783+0000 7f7ce4838200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:21.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:20.879+0000 7f7ce4838200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:35:21.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:21.709+0000 7f7ce4838200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:21.921+0000 7f7ce4838200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: Active manager daemon x restarted 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: Activating manager daemon x 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: osdmap e2: 0 total, 0 up, 0 in 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: mgrmap e6: x(active, starting, since 0.00351673s) 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:35:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: Manager daemon x is now available 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T23:35:22.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:21 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T23:35:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:22 smithi107 ceph-mon[29118]: Found migration_current of "None". Setting to last migration. 2024-04-24T23:35:23.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:22 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:23.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:22 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:23.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:22 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:23.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:22 smithi107 ceph-mon[29118]: mgrmap e7: x(active, since 1.00598s) 2024-04-24T23:35:23.560 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5062edb700 1 Processor -- start 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5062edb700 1 -- start start 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5062edb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5062edb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f505c106960 con 0x7f505c105f70 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5060c77700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5060c77700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54762/0 (socket says 172.21.15.107:54762) 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5060c77700 1 -- 172.21.15.107:0/383460536 learned_addr learned my addr 172.21.15.107:0/383460536 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:23.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.338+0000 7f5060c77700 1 -- 172.21.15.107:0/383460536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c106ae0 con 0x7f505c105f70 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f5060c77700 1 --2- 172.21.15.107:0/383460536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f50500098a0 tx=0x7f5050009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd15b006615b983c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f505b7fe700 1 -- 172.21.15.107:0/383460536 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5050004160 con 0x7f505c105f70 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f505b7fe700 1 -- 172.21.15.107:0/383460536 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5050004300 con 0x7f505c105f70 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f505b7fe700 1 -- 172.21.15.107:0/383460536 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5050011b50 con 0x7f505c105f70 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f5062edb700 1 -- 172.21.15.107:0/383460536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 msgr2=0x7f505c106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.339+0000 7f5062edb700 1 --2- 172.21.15.107:0/383460536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f50500098a0 tx=0x7f5050009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- 172.21.15.107:0/383460536 shutdown_connections 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 --2- 172.21.15.107:0/383460536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c106390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- 172.21.15.107:0/383460536 >> 172.21.15.107:0/383460536 conn(0x7f505c1018f0 msgr2=0x7f505c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- 172.21.15.107:0/383460536 shutdown_connections 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- 172.21.15.107:0/383460536 wait complete. 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 Processor -- start 2024-04-24T23:35:23.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- start start 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.340+0000 7f5062edb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f505c109e30 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5060c77700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5060c77700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54774/0 (socket says 172.21.15.107:54774) 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5060c77700 1 -- 172.21.15.107:0/3407077669 learned_addr learned my addr 172.21.15.107:0/3407077669 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5060c77700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c108db0 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5060c77700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5050009fb0 tx=0x7f5050011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50500038b0 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505c106fa0 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5050011e50 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5050019550 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.341+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f505c1074e0 con 0x7f505c105f70 2024-04-24T23:35:23.570 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.342+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f5050019850 con 0x7f505c105f70 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.342+0000 7f5059ffb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.342+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f505c063890 con 0x7f504c039000 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.342+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f505004e1e0 con 0x7f505c105f70 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.342+0000 7f505bfff700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.343+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.544+0000 7f505bfff700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.544+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.945+0000 7f505bfff700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:17.945+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:18.746+0000 7f505bfff700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:18.746+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:20.347+0000 7f505bfff700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1957718163 2024-04-24T23:35:23.571 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:20.347+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:21.925+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f505002d960 con 0x7f505c105f70 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:21.925+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 msgr2=0x7f504c03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:21.925+0000 7f5059ffb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.926+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f505004f1d0 con 0x7f505c105f70 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.927+0000 7f5059ffb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 0x7f504c03eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.927+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5050020e50 con 0x7f504c03c740 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.927+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 0x7f504c03eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.928+0000 7f505bfff700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 0x7f504c03eb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5048003a10 tx=0x7f5048009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.928+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f5050020e50 con 0x7f504c03c740 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.935+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f505c102bb0 con 0x7f504c03c740 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.935+0000 7f5059ffb700 1 -- 172.21.15.107:0/3407077669 <== mgr.14120 v2:172.21.15.107:6800/1738433151 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f505c102bb0 con 0x7f504c03c740 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 msgr2=0x7f504c03eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:23.572 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 0x7f504c03eb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5048003a10 tx=0x7f5048009290 comp rx=0 tx=0).stop 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 msgr2=0x7f505c108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5050009fb0 tx=0x7f5050011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 shutdown_connections 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f504c03c740 0x7f504c03eb30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f505c105f70 0x7f505c108870 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 --2- 172.21.15.107:0/3407077669 >> [v2:172.21.15.107:6800/1957718163,v1:172.21.15.107:6801/1957718163] conn(0x7f504c039000 0x7f504c03b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 >> 172.21.15.107:0/3407077669 conn(0x7f505c1018f0 msgr2=0x7f505c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 shutdown_connections 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:22.936+0000 7f5062edb700 1 -- 172.21.15.107:0/3407077669 wait complete. 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 5 is available 2024-04-24T23:35:23.573 INFO:teuthology.orchestra.run.smithi107.stdout:Setting orchestrator backend to cephadm... 2024-04-24T23:35:24.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:22] ENGINE Bus STARTING 2024-04-24T23:35:24.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:22] ENGINE Serving on https://172.21.15.107:7150 2024-04-24T23:35:24.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:22] ENGINE Serving on http://172.21.15.107:8765 2024-04-24T23:35:24.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:22] ENGINE Bus STARTED 2024-04-24T23:35:24.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T23:35:24.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:23 smithi107 ceph-mon[29118]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8c3c0700 1 Processor -- start 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8c3c0700 1 -- start start 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8c3c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8c3c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4d84108690 con 0x7f4d84107ca0 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8a15c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8a15c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54832/0 (socket says 172.21.15.107:54832) 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:23.999+0000 7f4d8a15c700 1 -- 172.21.15.107:0/2736908166 learned_addr learned my addr 172.21.15.107:0/2736908166 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8a15c700 1 -- 172.21.15.107:0/2736908166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d84108e50 con 0x7f4d84107ca0 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8a15c700 1 --2- 172.21.15.107:0/2736908166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098a0 tx=0x7f4d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecef4a3270270f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8915a700 1 -- 172.21.15.107:0/2736908166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d80004160 con 0x7f4d84107ca0 2024-04-24T23:35:24.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8915a700 1 -- 172.21.15.107:0/2736908166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d80004300 con 0x7f4d84107ca0 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8915a700 1 -- 172.21.15.107:0/2736908166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d80011b50 con 0x7f4d84107ca0 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/2736908166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 msgr2=0x7f4d841080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.000+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/2736908166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098a0 tx=0x7f4d80009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/2736908166 shutdown_connections 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/2736908166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d841080c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/2736908166 >> 172.21.15.107:0/2736908166 conn(0x7f4d841032a0 msgr2=0x7f4d841056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/2736908166 shutdown_connections 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/2736908166 wait complete. 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 Processor -- start 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- start start 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8c3c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8410c200 con 0x7f4d84107ca0 2024-04-24T23:35:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8a15c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8a15c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54844/0 (socket says 172.21.15.107:54844) 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.001+0000 7f4d8a15c700 1 -- 172.21.15.107:0/1215961501 learned_addr learned my addr 172.21.15.107:0/1215961501 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d8a15c700 1 -- 172.21.15.107:0/1215961501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8419c1d0 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d8a15c700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d80011850 tx=0x7f4d80011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d80003af0 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8419c460 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d80020070 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d800195a0 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.002+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8419f150 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.003+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4d8001a430 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.003+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d84063890 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.003+0000 7f4d7b7fe700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 0x7f4d7003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.003+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4d80011e00 con 0x7f4d84107ca0 2024-04-24T23:35:24.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.003+0000 7f4d8995b700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 0x7f4d7003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.004+0000 7f4d8995b700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 0x7f4d7003b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d74006fd0 tx=0x7f4d74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.008+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d8000e260 con 0x7f4d84107ca0 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.219+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4d84104560 con 0x7f4d700390a0 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.226+0000 7f4d7b7fe700 1 -- 172.21.15.107:0/1215961501 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4d84104560 con 0x7f4d700390a0 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.228+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 msgr2=0x7f4d7003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.228+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 0x7f4d7003b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d74006fd0 tx=0x7f4d74006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.228+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 msgr2=0x7f4d8419bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d80011850 tx=0x7f4d80011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 shutdown_connections 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4d700390a0 0x7f4d7003b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 --2- 172.21.15.107:0/1215961501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d84107ca0 0x7f4d8419bc90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 >> 172.21.15.107:0/1215961501 conn(0x7f4d841032a0 msgr2=0x7f4d84103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:24.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 shutdown_connections 2024-04-24T23:35:24.758 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:24.229+0000 7f4d8c3c0700 1 -- 172.21.15.107:0/1215961501 wait complete. 2024-04-24T23:35:25.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:24 smithi107 ceph-mon[29118]: mgrmap e8: x(active, since 2s) 2024-04-24T23:35:25.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:24 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:25.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:24 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:25.962 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-24T23:35:25.962 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.138+0000 7fe7e4eaa700 1 Processor -- start 2024-04-24T23:35:25.962 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.138+0000 7fe7e4eaa700 1 -- start start 2024-04-24T23:35:25.962 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7e4eaa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7e4eaa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e007c850 con 0x7fe7e007de20 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7de59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7de59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54856/0 (socket says 172.21.15.107:54856) 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7de59c700 1 -- 172.21.15.107:0/233420601 learned_addr learned my addr 172.21.15.107:0/233420601 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7de59c700 1 -- 172.21.15.107:0/233420601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e007c9d0 con 0x7fe7e007de20 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7de59c700 1 --2- 172.21.15.107:0/233420601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00098a0 tx=0x7fe7d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366f5225b337ec1f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:25.963 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7dd59a700 1 -- 172.21.15.107:0/233420601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7d0004160 con 0x7fe7e007de20 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7dd59a700 1 -- 172.21.15.107:0/233420601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7d0004300 con 0x7fe7e007de20 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.139+0000 7fe7dd59a700 1 -- 172.21.15.107:0/233420601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7d0011b50 con 0x7fe7e007de20 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/233420601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 msgr2=0x7fe7e007c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/233420601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00098a0 tx=0x7fe7d0009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/233420601 shutdown_connections 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/233420601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e007c280 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/233420601 >> 172.21.15.107:0/233420601 conn(0x7fe7e0103210 msgr2=0x7fe7e0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/233420601 shutdown_connections 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.140+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/233420601 wait complete. 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7e4eaa700 1 Processor -- start 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7e4eaa700 1 -- start start 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7e4eaa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:25.964 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7e4eaa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010e280 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7de59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7de59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54866/0 (socket says 172.21.15.107:54866) 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7de59c700 1 -- 172.21.15.107:0/3586207454 learned_addr learned my addr 172.21.15.107:0/3586207454 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7de59c700 1 -- 172.21.15.107:0/3586207454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e019e3e0 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7de59c700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009fb0 tx=0x7fe7d0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.141+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7d00038b0 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7d0011e50 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e019e670 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7d0019550 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e01a1360 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe7d0019850 con 0x7fe7e007de20 2024-04-24T23:35:25.965 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.142+0000 7fe7cf7fe700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 0x7fe7c803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.143+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe7d004d320 con 0x7fe7e007de20 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.143+0000 7fe7ddd9b700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 0x7fe7c803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.143+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e0063890 con 0x7fe7e007de20 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.143+0000 7fe7ddd9b700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 0x7fe7c803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006fd0 tx=0x7fe7d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.148+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7d0023c90 con 0x7fe7e007de20 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.345+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fe7e0104520 con 0x7fe7c8039000 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.346+0000 7fe7cf7fe700 1 -- 172.21.15.107:0/3586207454 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fe7e0104520 con 0x7fe7c8039000 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 msgr2=0x7fe7c803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 0x7fe7c803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006fd0 tx=0x7fe7d4006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 msgr2=0x7fe7e019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:25.966 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009fb0 tx=0x7fe7d0011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 shutdown_connections 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fe7c8039000 0x7fe7c803b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 --2- 172.21.15.107:0/3586207454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7e007de20 0x7fe7e019dea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 >> 172.21.15.107:0/3586207454 conn(0x7fe7e0103210 msgr2=0x7fe7e0103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 shutdown_connections 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:25.348+0000 7fe7e4eaa700 1 -- 172.21.15.107:0/3586207454 wait complete. 2024-04-24T23:35:25.967 INFO:teuthology.orchestra.run.smithi107.stdout:Generating ssh key... 2024-04-24T23:35:26.272 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:25 smithi107 ceph-mon[29118]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:26.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: Generating public/private rsa key pair. 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: Your identification has been saved in /tmp/tmp3c30ke14/key. 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: Your public key has been saved in /tmp/tmp3c30ke14/key.pub. 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: The key fingerprint is: 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: SHA256:g//d67rXRuCPXHUlXCSdAJUMRjVCdZN1EIPRRtLw8+I ceph-16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: The key's randomart image is: 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: +---[RSA 3072]----+ 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | oB@#OOO| 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . o=O==| 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | .o..| 2024-04-24T23:35:26.879 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . .oo| 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . S ...+| 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . . ...o| 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . .E* | 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . . .+ +| 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: | . +=+o | 2024-04-24T23:35:26.880 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: +----[SHA256]-----+ 2024-04-24T23:35:27.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-mon[29118]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:27.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:27.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:26 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:27.359 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f0888f35700 1 Processor -- start 2024-04-24T23:35:27.359 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f0888f35700 1 -- start start 2024-04-24T23:35:27.359 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f0888f35700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f0888f35700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0884106980 con 0x7f0884105f90 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f088259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f088259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54870/0 (socket says 172.21.15.107:54870) 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.375+0000 7f088259c700 1 -- 172.21.15.107:0/4198941907 learned_addr learned my addr 172.21.15.107:0/4198941907 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f088259c700 1 -- 172.21.15.107:0/4198941907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0884108fc0 con 0x7f0884105f90 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f088259c700 1 --2- 172.21.15.107:0/4198941907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08740098a0 tx=0x7f0874009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89be5ef53ba12aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f088159a700 1 -- 172.21.15.107:0/4198941907 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0874004160 con 0x7f0884105f90 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f088159a700 1 -- 172.21.15.107:0/4198941907 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0874004300 con 0x7f0884105f90 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f088159a700 1 -- 172.21.15.107:0/4198941907 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0874011b50 con 0x7f0884105f90 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f0888f35700 1 -- 172.21.15.107:0/4198941907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 msgr2=0x7f08841063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.376+0000 7f0888f35700 1 --2- 172.21.15.107:0/4198941907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08740098a0 tx=0x7f0874009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:27.360 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 -- 172.21.15.107:0/4198941907 shutdown_connections 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 --2- 172.21.15.107:0/4198941907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f08841063b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 -- 172.21.15.107:0/4198941907 >> 172.21.15.107:0/4198941907 conn(0x7f0884101940 msgr2=0x7f0884103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 -- 172.21.15.107:0/4198941907 shutdown_connections 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 -- 172.21.15.107:0/4198941907 wait complete. 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 Processor -- start 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.377+0000 7f0888f35700 1 -- start start 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f0888f35700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f0888f35700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f088410c1f0 con 0x7f0884105f90 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f088259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f088259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54884/0 (socket says 172.21.15.107:54884) 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f088259c700 1 -- 172.21.15.107:0/3209502266 learned_addr learned my addr 172.21.15.107:0/3209502266 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f088259c700 1 -- 172.21.15.107:0/3209502266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0884108d80 con 0x7f0884105f90 2024-04-24T23:35:27.361 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f088259c700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0874011850 tx=0x7f0874011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0874003af0 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0884106f70 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0874020070 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08740195a0 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.378+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08841074b0 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.379+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f08740198c0 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.379+0000 7f08737fe700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 0x7f086c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.379+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0874011e00 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.379+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0884063890 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.380+0000 7f0881d9b700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 0x7f086c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.380+0000 7f0881d9b700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 0x7f086c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0878006fd0 tx=0x7f0878006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.385+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0874003c90 con 0x7f0884105f90 2024-04-24T23:35:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.580+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f0884107cc0 con 0x7f086c039050 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.762+0000 7f08737fe700 1 -- 172.21.15.107:0/3209502266 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0884107cc0 con 0x7f086c039050 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 msgr2=0x7f086c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 0x7f086c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0878006fd0 tx=0x7f0878006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 msgr2=0x7f0884108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0874011850 tx=0x7f0874011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 shutdown_connections 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f086c039050 0x7f086c03b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 --2- 172.21.15.107:0/3209502266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0884105f90 0x7f0884108840 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.764+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 >> 172.21.15.107:0/3209502266 conn(0x7f0884101940 msgr2=0x7f08841024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.765+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 shutdown_connections 2024-04-24T23:35:27.363 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:26.765+0000 7f0888f35700 1 -- 172.21.15.107:0/3209502266 wait complete. 2024-04-24T23:35:28.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:27 smithi107 ceph-mon[29118]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:28.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:27 smithi107 ceph-mon[29118]: Generating ssh key... 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.753+0000 7f45f83c2700 1 Processor -- start 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.753+0000 7f45f83c2700 1 -- start start 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f83c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f83c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f45f007bdf0 con 0x7f45f0108740 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f615e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f615e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44930/0 (socket says 172.21.15.107:44930) 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f615e700 1 -- 172.21.15.107:0/2110719167 learned_addr learned my addr 172.21.15.107:0/2110719167 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f615e700 1 -- 172.21.15.107:0/2110719167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f010b070 con 0x7f45f0108740 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f615e700 1 --2- 172.21.15.107:0/2110719167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45e80098a0 tx=0x7f45e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fee7f56e67e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:28.616 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f515c700 1 -- 172.21.15.107:0/2110719167 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45e8004160 con 0x7f45f0108740 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f515c700 1 -- 172.21.15.107:0/2110719167 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f45e8004300 con 0x7f45f0108740 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.754+0000 7f45f515c700 1 -- 172.21.15.107:0/2110719167 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45e8011b50 con 0x7f45f0108740 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 -- 172.21.15.107:0/2110719167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 msgr2=0x7f45f010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2110719167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45e80098a0 tx=0x7f45e8009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 -- 172.21.15.107:0/2110719167 shutdown_connections 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2110719167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f010ab30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 -- 172.21.15.107:0/2110719167 >> 172.21.15.107:0/2110719167 conn(0x7f45f0102110 msgr2=0x7f45f0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 -- 172.21.15.107:0/2110719167 shutdown_connections 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.755+0000 7f45f83c2700 1 -- 172.21.15.107:0/2110719167 wait complete. 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f83c2700 1 Processor -- start 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f83c2700 1 -- start start 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f83c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f83c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f45f010c090 con 0x7f45f0108740 2024-04-24T23:35:28.617 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f615e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f615e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44938/0 (socket says 172.21.15.107:44938) 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f615e700 1 -- 172.21.15.107:0/2355282175 learned_addr learned my addr 172.21.15.107:0/2355282175 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f615e700 1 -- 172.21.15.107:0/2355282175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f019c180 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f615e700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f45e8009fb0 tx=0x7f45e8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45e80038b0 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f45e8011e50 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.756+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f019c410 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.757+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45e8019420 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.757+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f019f100 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.757+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f45e8019740 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.757+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f0063890 con 0x7f45f0108740 2024-04-24T23:35:28.618 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.757+0000 7f45e77fe700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 0x7f45dc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.758+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f45e804d440 con 0x7f45f0108740 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.758+0000 7f45f595d700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 0x7f45dc03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.758+0000 7f45f595d700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 0x7f45dc03b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.762+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f45e8018dc0 con 0x7f45f0108740 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.960+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f45f0103330 con 0x7f45dc039000 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.961+0000 7f45e77fe700 1 -- 172.21.15.107:0/2355282175 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f45f0103330 con 0x7f45dc039000 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 msgr2=0x7f45dc03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 0x7f45dc03b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 msgr2=0x7f45f019bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f45e8009fb0 tx=0x7f45e8011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 shutdown_connections 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f45dc039000 0x7f45dc03b4c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 --2- 172.21.15.107:0/2355282175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45f0108740 0x7f45f019bc40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 >> 172.21.15.107:0/2355282175 conn(0x7f45f0102110 msgr2=0x7f45f0102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 shutdown_connections 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:27.963+0000 7f45f83c2700 1 -- 172.21.15.107:0/2355282175 wait complete. 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:Adding key to root@localhost authorized_keys... 2024-04-24T23:35:28.620 INFO:teuthology.orchestra.run.smithi107.stdout:Adding host smithi107... 2024-04-24T23:35:29.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:28 smithi107 ceph-mon[29118]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:30.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:30 smithi107 ceph-mon[29118]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:30.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:30 smithi107 ceph-mon[29118]: Deploying cephadm binary to smithi107 2024-04-24T23:35:33.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Added host 'smithi107' with addr '172.21.15.107' 2024-04-24T23:35:33.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31f646d700 1 Processor -- start 2024-04-24T23:35:33.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31f646d700 1 -- start start 2024-04-24T23:35:33.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31f646d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31f646d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0108690 con 0x7f31f0107ca0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44950/0 (socket says 172.21.15.107:44950) 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31effff700 1 -- 172.21.15.107:0/539811543 learned_addr learned my addr 172.21.15.107:0/539811543 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31effff700 1 -- 172.21.15.107:0/539811543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f0108e50 con 0x7f31f0107ca0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.026+0000 7f31effff700 1 --2- 172.21.15.107:0/539811543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f31e00098a0 tx=0x7f31e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193de689f8e06fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31eeffd700 1 -- 172.21.15.107:0/539811543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0004160 con 0x7f31f0107ca0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31eeffd700 1 -- 172.21.15.107:0/539811543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31e0004300 con 0x7f31f0107ca0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31eeffd700 1 -- 172.21.15.107:0/539811543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0011b50 con 0x7f31f0107ca0 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 -- 172.21.15.107:0/539811543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 msgr2=0x7f31f01080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 --2- 172.21.15.107:0/539811543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f31e00098a0 tx=0x7f31e0009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 -- 172.21.15.107:0/539811543 shutdown_connections 2024-04-24T23:35:33.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 --2- 172.21.15.107:0/539811543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f01080c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 -- 172.21.15.107:0/539811543 >> 172.21.15.107:0/539811543 conn(0x7f31f01032a0 msgr2=0x7f31f01056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 -- 172.21.15.107:0/539811543 shutdown_connections 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.027+0000 7f31f646d700 1 -- 172.21.15.107:0/539811543 wait complete. 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31f646d700 1 Processor -- start 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31f646d700 1 -- start start 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31f646d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31f646d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31f010c200 con 0x7f31f0107ca0 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44966/0 (socket says 172.21.15.107:44966) 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31effff700 1 -- 172.21.15.107:0/2399829537 learned_addr learned my addr 172.21.15.107:0/2399829537 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.028+0000 7f31effff700 1 -- 172.21.15.107:0/2399829537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f019c1d0 con 0x7f31f0107ca0 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31effff700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fb0 tx=0x7f31e0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:33.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e00038b0 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f019c460 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31e0011e50 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0019550 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.029+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f019f150 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f31e0019850 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f0063890 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31ed7fa700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 0x7f31d803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f31e004d440 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31ef7fe700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 0x7f31d803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.030+0000 7f31ef7fe700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 0x7f31d803b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f31e4006fd0 tx=0x7f31e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.035+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31e0003ac0 con 0x7f31f0107ca0 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:29.233+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}) -- 0x7f31f0104560 con 0x7f31d8039000 2024-04-24T23:35:33.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.277+0000 7f31ed7fa700 1 -- 172.21.15.107:0/2399829537 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f31f0104560 con 0x7f31d8039000 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 msgr2=0x7f31d803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 0x7f31d803b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f31e4006fd0 tx=0x7f31e4006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 msgr2=0x7f31f019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fb0 tx=0x7f31e0011bf0 comp rx=0 tx=0).stop 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 shutdown_connections 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f31d8039000 0x7f31d803b4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 --2- 172.21.15.107:0/2399829537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f0107ca0 0x7f31f019bc90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 >> 172.21.15.107:0/2399829537 conn(0x7f31f01032a0 msgr2=0x7f31f0103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 shutdown_connections 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:33.279+0000 7f31f646d700 1 -- 172.21.15.107:0/2399829537 wait complete. 2024-04-24T23:35:33.886 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mon service... 2024-04-24T23:35:34.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:34 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:34.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:34 smithi107 ceph-mon[29118]: Added host smithi107 2024-04-24T23:35:34.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:34 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:35.108 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-24T23:35:35.108 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.288+0000 7f0f694b0700 1 Processor -- start 2024-04-24T23:35:35.108 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.288+0000 7f0f694b0700 1 -- start start 2024-04-24T23:35:35.108 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.288+0000 7f0f694b0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.288+0000 7f0f694b0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f64108690 con 0x7f0f64107ca0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f62ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f62ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44972/0 (socket says 172.21.15.107:44972) 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f62ffd700 1 -- 172.21.15.107:0/1029207948 learned_addr learned my addr 172.21.15.107:0/1029207948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f62ffd700 1 -- 172.21.15.107:0/1029207948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f64108e50 con 0x7f0f64107ca0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f62ffd700 1 --2- 172.21.15.107:0/1029207948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009e80 tx=0x7f0f5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1fa35cbdc1401d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f61ffb700 1 -- 172.21.15.107:0/1029207948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f5c017070 con 0x7f0f64107ca0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f61ffb700 1 -- 172.21.15.107:0/1029207948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f5c00b8c0 con 0x7f0f64107ca0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.289+0000 7f0f61ffb700 1 -- 172.21.15.107:0/1029207948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f5c0045c0 con 0x7f0f64107ca0 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 -- 172.21.15.107:0/1029207948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 msgr2=0x7f0f641080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1029207948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009e80 tx=0x7f0f5c00b330 comp rx=0 tx=0).stop 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 -- 172.21.15.107:0/1029207948 shutdown_connections 2024-04-24T23:35:35.109 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1029207948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f641080c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 -- 172.21.15.107:0/1029207948 >> 172.21.15.107:0/1029207948 conn(0x7f0f641032a0 msgr2=0x7f0f641056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 -- 172.21.15.107:0/1029207948 shutdown_connections 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 -- 172.21.15.107:0/1029207948 wait complete. 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.290+0000 7f0f694b0700 1 Processor -- start 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f694b0700 1 -- start start 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f694b0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f694b0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6410c200 con 0x7f0f64107ca0 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f62ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f62ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44978/0 (socket says 172.21.15.107:44978) 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f62ffd700 1 -- 172.21.15.107:0/1809406298 learned_addr learned my addr 172.21.15.107:0/1809406298 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f62ffd700 1 -- 172.21.15.107:0/1809406298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6419c1a0 con 0x7f0f64107ca0 2024-04-24T23:35:35.110 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f62ffd700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c004210 tx=0x7f0f5c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f5c020020 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f5c003b50 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f5c017040 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6419c430 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.291+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f6419f120 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.292+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0f5c023020 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.292+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f64063890 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.292+0000 7f0f5bfff700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 0x7f0f5003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.292+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f5c00f050 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.293+0000 7f0f627fc700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 0x7f0f5003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.293+0000 7f0f627fc700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 0x7f0f5003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c006fd0 tx=0x7f0f4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.297+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f5c02a020 con 0x7f0f64107ca0 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.509+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0f64104530 con 0x7f0f50039050 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.514+0000 7f0f5bfff700 1 -- 172.21.15.107:0/1809406298 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0f64104530 con 0x7f0f50039050 2024-04-24T23:35:35.111 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 msgr2=0x7f0f5003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 0x7f0f5003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c006fd0 tx=0x7f0f4c006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 msgr2=0x7f0f6419bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c004210 tx=0x7f0f5c00bd50 comp rx=0 tx=0).stop 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 shutdown_connections 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f0f50039050 0x7f0f5003b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 --2- 172.21.15.107:0/1809406298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f64107ca0 0x7f0f6419bc60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 >> 172.21.15.107:0/1809406298 conn(0x7f0f641032a0 msgr2=0x7f0f64103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 shutdown_connections 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:34.517+0000 7f0f694b0700 1 -- 172.21.15.107:0/1809406298 wait complete. 2024-04-24T23:35:35.112 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mgr service... 2024-04-24T23:35:35.856 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:35 smithi107 ceph-mon[29118]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:35.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:35 smithi107 ceph-mon[29118]: Saving service mon spec with placement count:5 2024-04-24T23:35:35.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:35 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:36.723 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-24T23:35:36.723 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bf4b24700 1 Processor -- start 2024-04-24T23:35:36.723 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bf4b24700 1 -- start start 2024-04-24T23:35:36.723 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bf4b24700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:36.723 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bf4b24700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf007be10 con 0x7f4bf0108720 2024-04-24T23:35:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44986/0 (socket says 172.21.15.107:44986) 2024-04-24T23:35:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.705+0000 7f4bee59c700 1 -- 172.21.15.107:0/522614832 learned_addr learned my addr 172.21.15.107:0/522614832 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bee59c700 1 -- 172.21.15.107:0/522614832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf010b050 con 0x7f4bf0108720 2024-04-24T23:35:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bee59c700 1 --2- 172.21.15.107:0/522614832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4547433dfe037795 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bed59a700 1 -- 172.21.15.107:0/522614832 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0004160 con 0x7f4bf0108720 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bed59a700 1 -- 172.21.15.107:0/522614832 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4be0004300 con 0x7f4bf0108720 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bed59a700 1 -- 172.21.15.107:0/522614832 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0011b50 con 0x7f4bf0108720 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bf4b24700 1 -- 172.21.15.107:0/522614832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 msgr2=0x7f4bf010ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.706+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/522614832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 -- 172.21.15.107:0/522614832 shutdown_connections 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/522614832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf010ab10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 -- 172.21.15.107:0/522614832 >> 172.21.15.107:0/522614832 conn(0x7f4bf0102170 msgr2=0x7f4bf0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 -- 172.21.15.107:0/522614832 shutdown_connections 2024-04-24T23:35:36.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 -- 172.21.15.107:0/522614832 wait complete. 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 Processor -- start 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.707+0000 7f4bf4b24700 1 -- start start 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bf4b24700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bf4b24700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010c000 con 0x7f4bf0108720 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:36.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44994/0 (socket says 172.21.15.107:44994) 2024-04-24T23:35:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bee59c700 1 -- 172.21.15.107:0/1587498413 learned_addr learned my addr 172.21.15.107:0/1587498413 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bee59c700 1 -- 172.21.15.107:0/1587498413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf019c140 con 0x7f4bf0108720 2024-04-24T23:35:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bee59c700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4be0011b60 tx=0x7f4be0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0003970 con 0x7f4bf0108720 2024-04-24T23:35:36.727 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4be0003b10 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf019c430 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0024610 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.708+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf019f0c0 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.709+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4be00248c0 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.709+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf0063890 con 0x7f4bf0108720 2024-04-24T23:35:36.728 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.709+0000 7f4bdf7fe700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 0x7f4bd803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.709+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4be0011e00 con 0x7f4bf0108720 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.709+0000 7f4bedd9b700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 0x7f4bd803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.710+0000 7f4bedd9b700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 0x7f4bd803b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4be4006fd0 tx=0x7f4be4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.714+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4be002d910 con 0x7f4bf0108720 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.927+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4bf01033e0 con 0x7f4bd80390a0 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.935+0000 7f4bdf7fe700 1 -- 172.21.15.107:0/1587498413 <== mgr.14120 v2:172.21.15.107:6800/1738433151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4bf01033e0 con 0x7f4bd80390a0 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 msgr2=0x7f4bd803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 0x7f4bd803b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4be4006fd0 tx=0x7f4be4006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 msgr2=0x7f4bf019bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4be0011b60 tx=0x7f4be0011c40 comp rx=0 tx=0).stop 2024-04-24T23:35:36.729 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 shutdown_connections 2024-04-24T23:35:36.730 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f4bd80390a0 0x7f4bd803b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:36.730 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 --2- 172.21.15.107:0/1587498413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0108720 0x7f4bf019bc00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:36.730 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 >> 172.21.15.107:0/1587498413 conn(0x7f4bf0102170 msgr2=0x7f4bf0102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:36.730 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 shutdown_connections 2024-04-24T23:35:36.730 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:35.938+0000 7f4bf4b24700 1 -- 172.21.15.107:0/1587498413 wait complete. 2024-04-24T23:35:37.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:36 smithi107 ceph-mon[29118]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:37.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:36 smithi107 ceph-mon[29118]: Saving service mgr spec with placement count:2 2024-04-24T23:35:37.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:36 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:38.029 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f15bba700 1 Processor -- start 2024-04-24T23:35:38.029 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f15bba700 1 -- start start 2024-04-24T23:35:38.029 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f15bba700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:38.029 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f15bba700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1007bdf0 con 0x7f7f10106540 2024-04-24T23:35:38.029 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53662/0 (socket says 172.21.15.107:53662) 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.126+0000 7f7f0f7fe700 1 -- 172.21.15.107:0/1000790409 learned_addr learned my addr 172.21.15.107:0/1000790409 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f0f7fe700 1 -- 172.21.15.107:0/1000790409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f10106ea0 con 0x7f7f10106540 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f0f7fe700 1 --2- 172.21.15.107:0/1000790409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f000098a0 tx=0x7f7f00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f17b3323d8e1ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f0effd700 1 -- 172.21.15.107:0/1000790409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f00004160 con 0x7f7f10106540 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f0effd700 1 -- 172.21.15.107:0/1000790409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f00004300 con 0x7f7f10106540 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f0effd700 1 -- 172.21.15.107:0/1000790409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f00011b50 con 0x7f7f10106540 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f15bba700 1 -- 172.21.15.107:0/1000790409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 msgr2=0x7f7f10106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.127+0000 7f7f15bba700 1 --2- 172.21.15.107:0/1000790409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f000098a0 tx=0x7f7f00009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:38.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- 172.21.15.107:0/1000790409 shutdown_connections 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 --2- 172.21.15.107:0/1000790409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f10106960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- 172.21.15.107:0/1000790409 >> 172.21.15.107:0/1000790409 conn(0x7f7f10102160 msgr2=0x7f7f10104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- 172.21.15.107:0/1000790409 shutdown_connections 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- 172.21.15.107:0/1000790409 wait complete. 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 Processor -- start 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- start start 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.128+0000 7f7f15bba700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10107f30 con 0x7f7f10106540 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53668/0 (socket says 172.21.15.107:53668) 2024-04-24T23:35:38.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0f7fe700 1 -- 172.21.15.107:0/3621814416 learned_addr learned my addr 172.21.15.107:0/3621814416 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0f7fe700 1 -- 172.21.15.107:0/3621814416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1019e3c0 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0f7fe700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f00011b60 tx=0x7f7f00011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f00003970 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f00003b10 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1019e650 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f00024610 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.129+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f101a1340 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.130+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7f00024910 con 0x7f7f10106540 2024-04-24T23:35:38.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.130+0000 7f7f0d7fa700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 0x7f7efc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.130+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7f00011e00 con 0x7f7f10106540 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.130+0000 7f7f06ffd700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 0x7f7efc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.130+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f10063890 con 0x7f7f10106540 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.131+0000 7f7f06ffd700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 0x7f7efc03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8006fd0 tx=0x7f7ef8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.135+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7f0002c350 con 0x7f7f10106540 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.332+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7f1002dd60 con 0x7f7f10106540 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.334+0000 7f7f0d7fa700 1 -- 172.21.15.107:0/3621814416 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f7f000172b0 con 0x7f7f10106540 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.336+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 msgr2=0x7f7efc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.336+0000 7f7f15bba700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 0x7f7efc03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8006fd0 tx=0x7f7ef8006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.336+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 msgr2=0x7f7f1019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:38.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.336+0000 7f7f15bba700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f00011b60 tx=0x7f7f00011c40 comp rx=0 tx=0).stop 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 shutdown_connections 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7efc039050 0x7f7efc03b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 --2- 172.21.15.107:0/3621814416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f10106540 0x7f7f1019de80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 >> 172.21.15.107:0/3621814416 conn(0x7f7f10102160 msgr2=0x7f7f10194aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 shutdown_connections 2024-04-24T23:35:38.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:37.337+0000 7f7f15bba700 1 -- 172.21.15.107:0/3621814416 wait complete. 2024-04-24T23:35:38.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:38 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3621814416' entity='client.admin' 2024-04-24T23:35:38.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:38 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:38.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:38 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:39.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:39 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1742190911' entity='client.admin' 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba7675700 1 Processor -- start 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba7675700 1 -- start start 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba7675700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba7675700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0107580 con 0x7f7ba0106b90 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba5411700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba5411700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53670/0 (socket says 172.21.15.107:53670) 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.478+0000 7f7ba5411700 1 -- 172.21.15.107:0/3501810590 learned_addr learned my addr 172.21.15.107:0/3501810590 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:39.855 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7ba5411700 1 -- 172.21.15.107:0/3501810590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba0107d90 con 0x7f7ba0106b90 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7ba5411700 1 --2- 172.21.15.107:0/3501810590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0098a0 tx=0x7f7b9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a548b6b7fd6126 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7b97fff700 1 -- 172.21.15.107:0/3501810590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b9c004160 con 0x7f7ba0106b90 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7b97fff700 1 -- 172.21.15.107:0/3501810590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c004300 con 0x7f7ba0106b90 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7b97fff700 1 -- 172.21.15.107:0/3501810590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b9c011b20 con 0x7f7ba0106b90 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7ba7675700 1 -- 172.21.15.107:0/3501810590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 msgr2=0x7f7ba0106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.479+0000 7f7ba7675700 1 --2- 172.21.15.107:0/3501810590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0098a0 tx=0x7f7b9c009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- 172.21.15.107:0/3501810590 shutdown_connections 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 --2- 172.21.15.107:0/3501810590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba0106fb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- 172.21.15.107:0/3501810590 >> 172.21.15.107:0/3501810590 conn(0x7f7ba0102110 msgr2=0x7f7ba0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- 172.21.15.107:0/3501810590 shutdown_connections 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- 172.21.15.107:0/3501810590 wait complete. 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 Processor -- start 2024-04-24T23:35:39.856 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- start start 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.480+0000 7f7ba7675700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba010b140 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba5411700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba5411700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53672/0 (socket says 172.21.15.107:53672) 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba5411700 1 -- 172.21.15.107:0/1742190911 learned_addr learned my addr 172.21.15.107:0/1742190911 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba5411700 1 -- 172.21.15.107:0/1742190911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba019e970 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba5411700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c000c00 tx=0x7f7b9c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b9c003b60 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c01a910 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba019eb40 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b9c024930 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.481+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba019eee0 con 0x7f7ba0106b90 2024-04-24T23:35:39.857 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.482+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7b9c022020 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.482+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba0063890 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.482+0000 7f7b967fc700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 0x7f7b8c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.482+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b9c04d2a0 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.482+0000 7f7ba4c10700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 0x7f7b8c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.483+0000 7f7ba4c10700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 0x7f7b8c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006fd0 tx=0x7f7b90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.487+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b9c01ab70 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.707+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f7ba019f220 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.710+0000 7f7b967fc700 1 -- 172.21.15.107:0/1742190911 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f7b9c02d3f0 con 0x7f7ba0106b90 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 msgr2=0x7f7b8c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 0x7f7b8c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006fd0 tx=0x7f7b90006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 msgr2=0x7f7ba007b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c000c00 tx=0x7f7b9c011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:39.858 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 shutdown_connections 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f7b8c039050 0x7f7b8c03b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 --2- 172.21.15.107:0/1742190911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ba0106b90 0x7f7ba007b9d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 >> 172.21.15.107:0/1742190911 conn(0x7f7ba0102110 msgr2=0x7f7ba0102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 shutdown_connections 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:38.713+0000 7f7ba7675700 1 -- 172.21.15.107:0/1742190911 wait complete. 2024-04-24T23:35:39.859 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling the dashboard module... 2024-04-24T23:35:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-mon[29118]: from='mgr.14120 172.21.15.107:0/2624274343' entity='mgr.x' 2024-04-24T23:35:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1207192800' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T23:35:41.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: ignoring --setuser ceph since I am not root 2024-04-24T23:35:41.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: ignoring --setgroup ceph since I am not root 2024-04-24T23:35:41.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:41.305+0000 7f459cdd0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:35:41.754 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:41.496+0000 7f459cdd0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:41.754 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:41.597+0000 7f459cdd0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:35:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.319+0000 7ff612e33700 1 Processor -- start 2024-04-24T23:35:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.319+0000 7ff612e33700 1 -- start start 2024-04-24T23:35:41.761 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.319+0000 7ff612e33700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.319+0000 7ff612e33700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c07c830 con 0x7ff60c07de00 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff610bcf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff610bcf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53684/0 (socket says 172.21.15.107:53684) 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff610bcf700 1 -- 172.21.15.107:0/1169623469 learned_addr learned my addr 172.21.15.107:0/1169623469 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff610bcf700 1 -- 172.21.15.107:0/1169623469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c07c9b0 con 0x7ff60c07de00 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff610bcf700 1 --2- 172.21.15.107:0/1169623469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6000098a0 tx=0x7ff600009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a460ce460d8c2809 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff60b7fe700 1 -- 172.21.15.107:0/1169623469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff600004160 con 0x7ff60c07de00 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff60b7fe700 1 -- 172.21.15.107:0/1169623469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff600004300 con 0x7ff60c07de00 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.320+0000 7ff60b7fe700 1 -- 172.21.15.107:0/1169623469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff600011b20 con 0x7ff60c07de00 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 -- 172.21.15.107:0/1169623469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 msgr2=0x7ff60c07c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 --2- 172.21.15.107:0/1169623469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6000098a0 tx=0x7ff600009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 -- 172.21.15.107:0/1169623469 shutdown_connections 2024-04-24T23:35:41.762 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 --2- 172.21.15.107:0/1169623469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c07c260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 -- 172.21.15.107:0/1169623469 >> 172.21.15.107:0/1169623469 conn(0x7ff60c103210 msgr2=0x7ff60c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 -- 172.21.15.107:0/1169623469 shutdown_connections 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.321+0000 7ff612e33700 1 -- 172.21.15.107:0/1169623469 wait complete. 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff612e33700 1 Processor -- start 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff612e33700 1 -- start start 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff612e33700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff612e33700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c10e2f0 con 0x7ff60c07de00 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff610bcf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff610bcf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53694/0 (socket says 172.21.15.107:53694) 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff610bcf700 1 -- 172.21.15.107:0/1207192800 learned_addr learned my addr 172.21.15.107:0/1207192800 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff610bcf700 1 -- 172.21.15.107:0/1207192800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c19e300 con 0x7ff60c07de00 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff610bcf700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6000116a0 tx=0x7ff600011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff600003a40 con 0x7ff60c07de00 2024-04-24T23:35:41.763 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff600003be0 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.322+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60c19e590 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.323+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff60001a750 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.323+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60c19e9f0 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.324+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ff60001aa70 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.324+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff60c063890 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.324+0000 7ff609ffb700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 0x7ff5fc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.324+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff60004d730 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.324+0000 7ff60bfff700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 0x7ff5fc03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.325+0000 7ff60bfff700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 0x7ff5fc03b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8006fd0 tx=0x7ff5f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.329+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff600011e20 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:40.598+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7ff60c1a1420 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.128+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7ff600016df0 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.130+0000 7ff609ffb700 1 -- 172.21.15.107:0/1207192800 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7ff60002d3f0 con 0x7ff60c07de00 2024-04-24T23:35:41.764 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 msgr2=0x7ff5fc03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 0x7ff5fc03b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8006fd0 tx=0x7ff5f8006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 msgr2=0x7ff60c19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6000116a0 tx=0x7ff600011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 shutdown_connections 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7ff5fc039000 0x7ff5fc03b4c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 --2- 172.21.15.107:0/1207192800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c07de00 0x7ff60c19ddc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 >> 172.21.15.107:0/1207192800 conn(0x7ff60c103210 msgr2=0x7ff60c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.131+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 shutdown_connections 2024-04-24T23:35:41.765 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:41.132+0000 7ff612e33700 1 -- 172.21.15.107:0/1207192800 wait complete. 2024-04-24T23:35:42.009 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:41.912+0000 7f459cdd0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:35:42.009 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:41.997+0000 7f459cdd0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:35:42.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:42 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1207192800' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T23:35:42.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:42 smithi107 ceph-mon[29118]: mgrmap e9: x(active, since 19s) 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.176+0000 7fdc32d12700 1 Processor -- start 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.176+0000 7fdc32d12700 1 -- start start 2024-04-24T23:35:43.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc32d12700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc32d12700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c07bdf0 con 0x7fdc2c108740 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc30aae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc30aae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53726/0 (socket says 172.21.15.107:53726) 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc30aae700 1 -- 172.21.15.107:0/676755843 learned_addr learned my addr 172.21.15.107:0/676755843 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc30aae700 1 -- 172.21.15.107:0/676755843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c10b070 con 0x7fdc2c108740 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc30aae700 1 --2- 172.21.15.107:0/676755843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098a0 tx=0x7fdc20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68be68cab6ab9db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc2b7fe700 1 -- 172.21.15.107:0/676755843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20004160 con 0x7fdc2c108740 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.177+0000 7fdc2b7fe700 1 -- 172.21.15.107:0/676755843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc20004300 con 0x7fdc2c108740 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc2b7fe700 1 -- 172.21.15.107:0/676755843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20011b20 con 0x7fdc2c108740 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 -- 172.21.15.107:0/676755843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 msgr2=0x7fdc2c10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 --2- 172.21.15.107:0/676755843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098a0 tx=0x7fdc20009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 -- 172.21.15.107:0/676755843 shutdown_connections 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 --2- 172.21.15.107:0/676755843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c10ab30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 -- 172.21.15.107:0/676755843 >> 172.21.15.107:0/676755843 conn(0x7fdc2c102110 msgr2=0x7fdc2c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 -- 172.21.15.107:0/676755843 shutdown_connections 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.178+0000 7fdc32d12700 1 -- 172.21.15.107:0/676755843 wait complete. 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc32d12700 1 Processor -- start 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc32d12700 1 -- start start 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc32d12700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc32d12700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c10c0d0 con 0x7fdc2c108740 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc30aae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc30aae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53732/0 (socket says 172.21.15.107:53732) 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.179+0000 7fdc30aae700 1 -- 172.21.15.107:0/1091241814 learned_addr learned my addr 172.21.15.107:0/1091241814 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc30aae700 1 -- 172.21.15.107:0/1091241814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c19c100 con 0x7fdc2c108740 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc30aae700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc200116a0 tx=0x7fdc20011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc20003a40 con 0x7fdc2c108740 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c19c390 con 0x7fdc2c108740 2024-04-24T23:35:43.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc20003be0 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc2001a750 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.180+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c19c7f0 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.181+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c063890 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.181+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fdc2001aa50 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.182+0000 7fdc29ffb700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 0x7fdc1c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.182+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc2004d980 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.182+0000 7fdc2bfff700 1 -- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 msgr2=0x7fdc1c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.182+0000 7fdc2bfff700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 0x7fdc1c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.186+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc20011e20 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.383+0000 7fdc2bfff700 1 -- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 msgr2=0x7fdc1c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.383+0000 7fdc2bfff700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 0x7fdc1c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.479+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fdc2c19f260 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.480+0000 7fdc29ffb700 1 -- 172.21.15.107:0/1091241814 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fdc2000ba30 con 0x7fdc2c108740 2024-04-24T23:35:43.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 msgr2=0x7fdc1c03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 0x7fdc1c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 msgr2=0x7fdc2c19bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc200116a0 tx=0x7fdc20011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 shutdown_connections 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7fdc1c039050 0x7fdc1c03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 --2- 172.21.15.107:0/1091241814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c108740 0x7fdc2c19bbc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 >> 172.21.15.107:0/1091241814 conn(0x7fdc2c102110 msgr2=0x7fdc2c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.482+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 shutdown_connections 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:42.483+0000 7fdc32d12700 1 -- 172.21.15.107:0/1091241814 wait complete. 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-04-24T23:35:43.082 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 9... 2024-04-24T23:35:43.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1091241814' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T23:35:43.480 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:43.195+0000 7f459cdd0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:35:43.480 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:43.371+0000 7f459cdd0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:43.787 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:43.544+0000 7f459cdd0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:35:44.125 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:43.833+0000 7f459cdd0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:35:44.125 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:43.925+0000 7f459cdd0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:35:44.125 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:44.124+0000 7f459cdd0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:35:44.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:44.314+0000 7f459cdd0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:35:45.319 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:44.902+0000 7f459cdd0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:35:45.612 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:45.317+0000 7f459cdd0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:35:45.612 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:45.412+0000 7f459cdd0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:35:45.612 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:45.611+0000 7f459cdd0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:35:46.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.125+0000 7f459cdd0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:35:46.378 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.215+0000 7f459cdd0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:35:46.791 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.453+0000 7f459cdd0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:35:46.791 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.583+0000 7f459cdd0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:35:46.791 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.677+0000 7f459cdd0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:35:46.792 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.790+0000 7f459cdd0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:35:47.128 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:46.880+0000 7f459cdd0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:35:47.942 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:47.676+0000 7f459cdd0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:35:47.942 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:35:47.889+0000 7f459cdd0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: Active manager daemon x restarted 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: Activating manager daemon x 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: osdmap e3: 0 total, 0 up, 0 in 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: mgrmap e10: x(active, starting, since 0.00349209s) 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T23:35:48.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T23:35:48.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T23:35:48.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T23:35:48.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: Manager daemon x is now available 2024-04-24T23:35:48.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:35:48.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T23:35:49.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T23:35:49.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:48 smithi107 ceph-mon[29118]: mgrmap e11: x(active, since 1.00663s) 2024-04-24T23:35:49.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-04-24T23:35:49.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-24T23:35:49.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T23:35:49.831 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35e5f14700 1 Processor -- start 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35e5f14700 1 -- start start 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35e5f14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35e5f14700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35e0108660 con 0x7f35e0107c70 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53738/0 (socket says 172.21.15.107:53738) 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.580+0000 7f35df7fe700 1 -- 172.21.15.107:0/659788690 learned_addr learned my addr 172.21.15.107:0/659788690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.581+0000 7f35df7fe700 1 -- 172.21.15.107:0/659788690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e0108e70 con 0x7f35e0107c70 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.581+0000 7f35df7fe700 1 --2- 172.21.15.107:0/659788690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35d00098a0 tx=0x7f35d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae5b787f352549d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:49.832 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.581+0000 7f35de7fc700 1 -- 172.21.15.107:0/659788690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d0004160 con 0x7f35e0107c70 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.581+0000 7f35de7fc700 1 -- 172.21.15.107:0/659788690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d0004300 con 0x7f35e0107c70 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.581+0000 7f35de7fc700 1 -- 172.21.15.107:0/659788690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d0011b20 con 0x7f35e0107c70 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 -- 172.21.15.107:0/659788690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 msgr2=0x7f35e0108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 --2- 172.21.15.107:0/659788690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35d00098a0 tx=0x7f35d0009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 -- 172.21.15.107:0/659788690 shutdown_connections 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 --2- 172.21.15.107:0/659788690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e0108090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 -- 172.21.15.107:0/659788690 >> 172.21.15.107:0/659788690 conn(0x7f35e0103210 msgr2=0x7f35e0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 -- 172.21.15.107:0/659788690 shutdown_connections 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 -- 172.21.15.107:0/659788690 wait complete. 2024-04-24T23:35:49.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.582+0000 7f35e5f14700 1 Processor -- start 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35e5f14700 1 -- start start 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35e5f14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35e5f14700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010c220 con 0x7f35e0107c70 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53752/0 (socket says 172.21.15.107:53752) 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35df7fe700 1 -- 172.21.15.107:0/4101827541 learned_addr learned my addr 172.21.15.107:0/4101827541 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35df7fe700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e007e030 con 0x7f35e0107c70 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.583+0000 7f35df7fe700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f35d00116a0 tx=0x7f35d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.584+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d0003a40 con 0x7f35e0107c70 2024-04-24T23:35:49.834 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.584+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e007c220 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.584+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d0003be0 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.584+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d001a720 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.584+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e007c6e0 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f35d001aa40 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35dcff9700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35c803bc20 con 0x7f35c8039050 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f35d004d710 con 0x7f35e0107c70 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35deffd700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.585+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.786+0000 7f35deffd700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:49.835 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:43.786+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:44.187+0000 7f35deffd700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:44.187+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:44.988+0000 7f35deffd700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:44.988+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:46.589+0000 7f35deffd700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1738433151 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:46.589+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:47.892+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f35d002d910 con 0x7f35e0107c70 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:47.892+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 msgr2=0x7f35c803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:47.892+0000 7f35dcff9700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.894+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f35d004f740 con 0x7f35e0107c70 2024-04-24T23:35:49.836 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.895+0000 7f35dcff9700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 0x7f35c803ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.895+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35d0020ec0 con 0x7f35c803cb10 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.895+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 0x7f35c803ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.896+0000 7f35deffd700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 0x7f35c803ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f35d4003a10 tx=0x7f35d40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.897+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f35d0020ec0 con 0x7f35c803cb10 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f35e0063890 con 0x7f35c803cb10 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35dcff9700 1 -- 172.21.15.107:0/4101827541 <== mgr.14152 v2:172.21.15.107:6800/1159279165 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f35e0063890 con 0x7f35c803cb10 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 msgr2=0x7f35c803ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35e5f14700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 0x7f35c803ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f35d4003a10 tx=0x7f35d40092b0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 msgr2=0x7f35e007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.903+0000 7f35e5f14700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f35d00116a0 tx=0x7f35d0011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.837 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 shutdown_connections 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35c803cb10 0x7f35c803ef00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35e0107c70 0x7f35e007daf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 --2- 172.21.15.107:0/4101827541 >> [v2:172.21.15.107:6800/1738433151,v1:172.21.15.107:6801/1738433151] conn(0x7f35c8039050 0x7f35c803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 >> 172.21.15.107:0/4101827541 conn(0x7f35e0103210 msgr2=0x7f35e0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 shutdown_connections 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:48.904+0000 7f35e5f14700 1 -- 172.21.15.107:0/4101827541 wait complete. 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 9 is available 2024-04-24T23:35:49.838 INFO:teuthology.orchestra.run.smithi107.stdout:Generating a dashboard self-signed certificate... 2024-04-24T23:35:50.087 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:49 smithi107 ceph-mon[29118]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T23:35:50.087 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:49 smithi107 ceph-mon[29118]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T23:35:50.087 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:35:51.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:49] ENGINE Bus STARTING 2024-04-24T23:35:51.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:49] ENGINE Serving on http://172.21.15.107:8765 2024-04-24T23:35:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:49] ENGINE Serving on https://172.21.15.107:7150 2024-04-24T23:35:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: [24/Apr/2024:23:35:49] ENGINE Bus STARTED 2024-04-24T23:35:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: mgrmap e12: x(active, since 2s) 2024-04-24T23:35:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:35:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:35:51.304 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-24T23:35:51.304 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07aab65700 1 Processor -- start 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07aab65700 1 -- start start 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07aab65700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07aab65700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4109c80 con 0x7f07a4109290 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07a8901700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07a8901700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42300/0 (socket says 172.21.15.107:42300) 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.325+0000 7f07a8901700 1 -- 172.21.15.107:0/1218386149 learned_addr learned my addr 172.21.15.107:0/1218386149 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.326+0000 7f07a8901700 1 -- 172.21.15.107:0/1218386149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a4109e00 con 0x7f07a4109290 2024-04-24T23:35:51.305 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.326+0000 7f07a8901700 1 --2- 172.21.15.107:0/1218386149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f07980098a0 tx=0x7f0798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10583797ddf32c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.326+0000 7f07a37fe700 1 -- 172.21.15.107:0/1218386149 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0798004160 con 0x7f07a4109290 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.326+0000 7f07a37fe700 1 -- 172.21.15.107:0/1218386149 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0798004300 con 0x7f07a4109290 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.326+0000 7f07a37fe700 1 -- 172.21.15.107:0/1218386149 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0798011b20 con 0x7f07a4109290 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 -- 172.21.15.107:0/1218386149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 msgr2=0x7f07a41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 --2- 172.21.15.107:0/1218386149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f07980098a0 tx=0x7f0798009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 -- 172.21.15.107:0/1218386149 shutdown_connections 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 --2- 172.21.15.107:0/1218386149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a41096b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 -- 172.21.15.107:0/1218386149 >> 172.21.15.107:0/1218386149 conn(0x7f07a41018f0 msgr2=0x7f07a4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 -- 172.21.15.107:0/1218386149 shutdown_connections 2024-04-24T23:35:51.306 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.327+0000 7f07aab65700 1 -- 172.21.15.107:0/1218386149 wait complete. 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07aab65700 1 Processor -- start 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07aab65700 1 -- start start 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07aab65700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07aab65700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4106ba0 con 0x7f07a4109290 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07a8901700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07a8901700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42314/0 (socket says 172.21.15.107:42314) 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07a8901700 1 -- 172.21.15.107:0/1530024172 learned_addr learned my addr 172.21.15.107:0/1530024172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07a8901700 1 -- 172.21.15.107:0/1530024172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a419c100 con 0x7f07a4109290 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.328+0000 7f07a8901700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f07980116a0 tx=0x7f0798011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.329+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0798003a40 con 0x7f07a4109290 2024-04-24T23:35:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.329+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a419c3f0 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.329+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0798003be0 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.329+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f079801a720 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.329+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a419f080 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f079801aa20 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07a1ffb700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 0x7f079403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a4063890 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f079804d5a0 con 0x7f07a4109290 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07a3fff700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 0x7f079403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.330+0000 7f07a3fff700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 0x7f079403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0790006fd0 tx=0x7f0790006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:51.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.335+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0798011e20 con 0x7f07a4109290 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.497+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f0798024430 con 0x7f07a4109290 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.568+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f07a4102ad0 con 0x7f0794038f40 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.755+0000 7f07a1ffb700 1 -- 172.21.15.107:0/1530024172 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f07a4102ad0 con 0x7f0794038f40 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 msgr2=0x7f079403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 0x7f079403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0790006fd0 tx=0x7f0790006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 msgr2=0x7f07a419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f07980116a0 tx=0x7f0798011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 shutdown_connections 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0794038f40 0x7f079403b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 --2- 172.21.15.107:0/1530024172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4109290 0x7f07a419bbc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 >> 172.21.15.107:0/1530024172 conn(0x7f07a41018f0 msgr2=0x7f07a41023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:51.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 shutdown_connections 2024-04-24T23:35:51.310 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:50.758+0000 7f07aab65700 1 -- 172.21.15.107:0/1530024172 wait complete. 2024-04-24T23:35:51.310 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial admin user... 2024-04-24T23:35:52.199 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:51 smithi107 ceph-mon[29118]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:52.199 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:35:52.768 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zCj54Joa1ib6Pco13yHH5ei66NqP/02CLw6k9L.pmvqUPj05u0rTa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714001752, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-24T23:35:52.768 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.725+0000 7f489da72700 1 Processor -- start 2024-04-24T23:35:52.768 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.725+0000 7f489da72700 1 -- start start 2024-04-24T23:35:52.768 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f489da72700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:52.768 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f489da72700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4898108690 con 0x7f4898107ca0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42316/0 (socket says 172.21.15.107:42316) 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48977fe700 1 -- 172.21.15.107:0/3666551077 learned_addr learned my addr 172.21.15.107:0/3666551077 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48977fe700 1 -- 172.21.15.107:0/3666551077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4898108e50 con 0x7f4898107ca0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48977fe700 1 --2- 172.21.15.107:0/3666551077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48880098a0 tx=0x7f4888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85059c30529058c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.726+0000 7f48967fc700 1 -- 172.21.15.107:0/3666551077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4888004160 con 0x7f4898107ca0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f48967fc700 1 -- 172.21.15.107:0/3666551077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4888004300 con 0x7f4898107ca0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f48967fc700 1 -- 172.21.15.107:0/3666551077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4888011b20 con 0x7f4898107ca0 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 -- 172.21.15.107:0/3666551077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 msgr2=0x7f48981080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 --2- 172.21.15.107:0/3666551077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48880098a0 tx=0x7f4888009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:52.769 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 -- 172.21.15.107:0/3666551077 shutdown_connections 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 --2- 172.21.15.107:0/3666551077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f48981080c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 -- 172.21.15.107:0/3666551077 >> 172.21.15.107:0/3666551077 conn(0x7f48981032c0 msgr2=0x7f48981056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 -- 172.21.15.107:0/3666551077 shutdown_connections 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.727+0000 7f489da72700 1 -- 172.21.15.107:0/3666551077 wait complete. 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f489da72700 1 Processor -- start 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f489da72700 1 -- start start 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f489da72700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f489da72700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f489810c200 con 0x7f4898107ca0 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f48977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:52.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f48977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42330/0 (socket says 172.21.15.107:42330) 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f48977fe700 1 -- 172.21.15.107:0/2797433702 learned_addr learned my addr 172.21.15.107:0/2797433702 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.728+0000 7f48977fe700 1 -- 172.21.15.107:0/2797433702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489807e030 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f48977fe700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48880117e0 tx=0x7f4888011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4888003bf0 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f488801a910 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489807c220 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48880249a0 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.729+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f489807c6e0 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f4888022020 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4898063890 con 0x7f4898107ca0 2024-04-24T23:35:52.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f4894ff9700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 0x7f488003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f488804d470 con 0x7f4898107ca0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f4896ffd700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 0x7f488003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.730+0000 7f4896ffd700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 0x7f488003b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f488c006fd0 tx=0x7f488c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.735+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4888011e20 con 0x7f4898107ca0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:51.957+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f489807ce50 con 0x7f48800390b0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.185+0000 7f4894ff9700 1 -- 172.21.15.107:0/2797433702 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f489807ce50 con 0x7f48800390b0 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 msgr2=0x7f488003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 0x7f488003b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f488c006fd0 tx=0x7f488c006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 msgr2=0x7f489807daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48880117e0 tx=0x7f4888011c10 comp rx=0 tx=0).stop 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 shutdown_connections 2024-04-24T23:35:52.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48800390b0 0x7f488003b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:52.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 --2- 172.21.15.107:0/2797433702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4898107ca0 0x7f489807daf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:52.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 >> 172.21.15.107:0/2797433702 conn(0x7f48981032c0 msgr2=0x7f4898103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:52.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 shutdown_connections 2024-04-24T23:35:52.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:52.188+0000 7f489da72700 1 -- 172.21.15.107:0/2797433702 wait complete. 2024-04-24T23:35:52.773 INFO:teuthology.orchestra.run.smithi107.stdout:Fetching dashboard port number... 2024-04-24T23:35:53.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:53 smithi107 ceph-mon[29118]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:35:53.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:53 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:35:54.118 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 8443 2024-04-24T23:35:54.118 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170b891700 1 Processor -- start 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170b891700 1 -- start start 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170b891700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170b891700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1704106970 con 0x7f1704105f80 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170962d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170962d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42338/0 (socket says 172.21.15.107:42338) 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.231+0000 7f170962d700 1 -- 172.21.15.107:0/2927985794 learned_addr learned my addr 172.21.15.107:0/2927985794 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.232+0000 7f170962d700 1 -- 172.21.15.107:0/2927985794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1704108fb0 con 0x7f1704105f80 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.232+0000 7f170962d700 1 --2- 172.21.15.107:0/2927985794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17000098a0 tx=0x7f1700009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9919dcb57d6d158d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.232+0000 7f16fbfff700 1 -- 172.21.15.107:0/2927985794 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1700004160 con 0x7f1704105f80 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.232+0000 7f16fbfff700 1 -- 172.21.15.107:0/2927985794 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1700004300 con 0x7f1704105f80 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.232+0000 7f16fbfff700 1 -- 172.21.15.107:0/2927985794 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1700011b20 con 0x7f1704105f80 2024-04-24T23:35:54.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 -- 172.21.15.107:0/2927985794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 msgr2=0x7f17041063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 --2- 172.21.15.107:0/2927985794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17000098a0 tx=0x7f1700009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 -- 172.21.15.107:0/2927985794 shutdown_connections 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 --2- 172.21.15.107:0/2927985794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f17041063a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 -- 172.21.15.107:0/2927985794 >> 172.21.15.107:0/2927985794 conn(0x7f17041018f0 msgr2=0x7f1704103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 -- 172.21.15.107:0/2927985794 shutdown_connections 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 -- 172.21.15.107:0/2927985794 wait complete. 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.233+0000 7f170b891700 1 Processor -- start 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170b891700 1 -- start start 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170b891700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:54.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170b891700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f170410c1e0 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170962d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170962d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42348/0 (socket says 172.21.15.107:42348) 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170962d700 1 -- 172.21.15.107:0/2080080437 learned_addr learned my addr 172.21.15.107:0/2080080437 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170962d700 1 -- 172.21.15.107:0/2080080437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1704108d70 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f170962d700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17000116a0 tx=0x7f1700011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.234+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1700003a40 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.235+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1700003be0 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.235+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1704106f60 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.235+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f170001a720 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.235+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1704107420 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f170001a9d0 con 0x7f1704105f80 2024-04-24T23:35:54.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1704063890 con 0x7f1704105f80 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f16fa7fc700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 0x7f16f003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f170004d7d0 con 0x7f1704105f80 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f1708e2c700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 0x7f16f003b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.236+0000 7f1708e2c700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 0x7f16f003b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16f4006fd0 tx=0x7f16f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.241+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1700011e20 con 0x7f1704105f80 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.441+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f17041076a0 con 0x7f1704105f80 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.442+0000 7f16fa7fc700 1 -- 172.21.15.107:0/2080080437 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f170001adc0 con 0x7f1704105f80 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 msgr2=0x7f16f003b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 0x7f16f003b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16f4006fd0 tx=0x7f16f4006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 msgr2=0x7f1704108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17000116a0 tx=0x7f1700011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:54.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 shutdown_connections 2024-04-24T23:35:54.123 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f0039100 0x7f16f003b5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:54.123 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 --2- 172.21.15.107:0/2080080437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1704105f80 0x7f1704108830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:54.123 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 >> 172.21.15.107:0/2080080437 conn(0x7f17041018f0 msgr2=0x7f17041023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:54.123 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 shutdown_connections 2024-04-24T23:35:54.123 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:53.444+0000 7f170b891700 1 -- 172.21.15.107:0/2080080437 wait complete. 2024-04-24T23:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T23:35:54.126 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-04-24T23:35:54.132 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T23:35:54.133 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-04-24T23:35:54.133 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-04-24T23:35:54.133 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-24T23:35:54.134 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph Dashboard is now available at: 2024-04-24T23:35:54.134 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:54.134 INFO:teuthology.orchestra.run.smithi107.stdout: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-04-24T23:35:54.134 INFO:teuthology.orchestra.run.smithi107.stdout: User: admin 2024-04-24T23:35:54.134 INFO:teuthology.orchestra.run.smithi107.stdout: Password: jiklpg4hfi 2024-04-24T23:35:54.135 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:54.135 INFO:teuthology.orchestra.run.smithi107.stdout:Saving cluster configuration to /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config directory 2024-04-24T23:35:54.135 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling autotune for osd_memory_target 2024-04-24T23:35:54.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:54 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2080080437' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T23:35:55.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:55 smithi107 ceph-mon[29118]: mgrmap e13: x(active, since 6s) 2024-04-24T23:35:56.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.291+0000 7fdeaa756700 1 Processor -- start 2024-04-24T23:35:56.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.291+0000 7fdeaa756700 1 -- start start 2024-04-24T23:35:56.112 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdeaa756700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdeaa756700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdea407bdb0 con 0x7fdea4108780 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42352/0 (socket says 172.21.15.107:42352) 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea3fff700 1 -- 172.21.15.107:0/3400655785 learned_addr learned my addr 172.21.15.107:0/3400655785 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea3fff700 1 -- 172.21.15.107:0/3400655785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea410b0b0 con 0x7fdea4108780 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea3fff700 1 --2- 172.21.15.107:0/3400655785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde940098a0 tx=0x7fde94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790fe91da2e58119 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea2ffd700 1 -- 172.21.15.107:0/3400655785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94004160 con 0x7fdea4108780 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.292+0000 7fdea2ffd700 1 -- 172.21.15.107:0/3400655785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94004300 con 0x7fdea4108780 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdea2ffd700 1 -- 172.21.15.107:0/3400655785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94011b20 con 0x7fdea4108780 2024-04-24T23:35:56.113 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 -- 172.21.15.107:0/3400655785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 msgr2=0x7fdea410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3400655785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde940098a0 tx=0x7fde94009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 -- 172.21.15.107:0/3400655785 shutdown_connections 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3400655785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea410ab70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 -- 172.21.15.107:0/3400655785 >> 172.21.15.107:0/3400655785 conn(0x7fdea4102170 msgr2=0x7fdea4104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 -- 172.21.15.107:0/3400655785 shutdown_connections 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.293+0000 7fdeaa756700 1 -- 172.21.15.107:0/3400655785 wait complete. 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdeaa756700 1 Processor -- start 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdeaa756700 1 -- start start 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdeaa756700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdeaa756700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdea410c0b0 con 0x7fdea4108780 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42362/0 (socket says 172.21.15.107:42362) 2024-04-24T23:35:56.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdea3fff700 1 -- 172.21.15.107:0/3790842959 learned_addr learned my addr 172.21.15.107:0/3790842959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.294+0000 7fdea3fff700 1 -- 172.21.15.107:0/3790842959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea4108540 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdea3fff700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde94011b60 tx=0x7fde94011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94003b60 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9401a910 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94024930 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea4106730 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.295+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea4106bf0 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.296+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea4063890 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.296+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fde94022020 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.296+0000 7fdea17fa700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 0x7fde8c03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.296+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fde9404d440 con 0x7fdea4108780 2024-04-24T23:35:56.115 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.296+0000 7fdea37fe700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 0x7fde8c03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.297+0000 7fdea37fe700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 0x7fde8c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde98006fd0 tx=0x7fde98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.301+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde9400b3e0 con 0x7fdea4108780 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.497+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fdea4106ee0 con 0x7fdea4108780 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.497+0000 7fdea17fa700 1 -- 172.21.15.107:0/3790842959 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fdea4106ee0 con 0x7fdea4108780 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 msgr2=0x7fde8c03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 0x7fde8c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde98006fd0 tx=0x7fde98006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 msgr2=0x7fdea4108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde94011b60 tx=0x7fde94011c40 comp rx=0 tx=0).stop 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 shutdown_connections 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde8c0391a0 0x7fde8c03b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 --2- 172.21.15.107:0/3790842959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdea4108780 0x7fdea4108000 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:56.116 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.499+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 >> 172.21.15.107:0/3790842959 conn(0x7fdea4102170 msgr2=0x7fdea4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:56.117 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.500+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 shutdown_connections 2024-04-24T23:35:56.117 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:55.500+0000 7fdeaa756700 1 -- 172.21.15.107:0/3790842959 wait complete. 2024-04-24T23:35:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.524+0000 7f61a6bc3700 1 Processor -- start 2024-04-24T23:35:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.524+0000 7f61a6bc3700 1 -- start start 2024-04-24T23:35:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a6bc3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a6bc3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61a007be10 con 0x7f61a0108740 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a495f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a495f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52994/0 (socket says 172.21.15.107:52994) 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a495f700 1 -- 172.21.15.107:0/104572486 learned_addr learned my addr 172.21.15.107:0/104572486 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a495f700 1 -- 172.21.15.107:0/104572486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a010b070 con 0x7f61a0108740 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f61a495f700 1 --2- 172.21.15.107:0/104572486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61940098a0 tx=0x7f6194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961bd12220113f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f619f7fe700 1 -- 172.21.15.107:0/104572486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6194004160 con 0x7f61a0108740 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f619f7fe700 1 -- 172.21.15.107:0/104572486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6194004300 con 0x7f61a0108740 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.525+0000 7f619f7fe700 1 -- 172.21.15.107:0/104572486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6194011b20 con 0x7f61a0108740 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 -- 172.21.15.107:0/104572486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 msgr2=0x7f61a010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/104572486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61940098a0 tx=0x7f6194009bb0 comp rx=0 tx=0).stop 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 -- 172.21.15.107:0/104572486 shutdown_connections 2024-04-24T23:35:57.379 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/104572486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a010ab30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 -- 172.21.15.107:0/104572486 >> 172.21.15.107:0/104572486 conn(0x7f61a0102170 msgr2=0x7f61a0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 -- 172.21.15.107:0/104572486 shutdown_connections 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.526+0000 7f61a6bc3700 1 -- 172.21.15.107:0/104572486 wait complete. 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a6bc3700 1 Processor -- start 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a6bc3700 1 -- start start 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a6bc3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a6bc3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61a010c0d0 con 0x7f61a0108740 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a495f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a495f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53002/0 (socket says 172.21.15.107:53002) 2024-04-24T23:35:57.380 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a495f700 1 -- 172.21.15.107:0/1476455890 learned_addr learned my addr 172.21.15.107:0/1476455890 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a495f700 1 -- 172.21.15.107:0/1476455890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a019c100 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a495f700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f61940116a0 tx=0x7f6194011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6194003a40 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a019c3f0 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6194003be0 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.527+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f619401a720 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.528+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a019f080 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.528+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f6194024aa0 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.529+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a0063890 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.529+0000 7f619dffb700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 0x7f619003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.529+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f619404d8f0 con 0x7f61a0108740 2024-04-24T23:35:57.381 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.529+0000 7f619ffff700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 0x7f619003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.529+0000 7f619ffff700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 0x7f619003b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.534+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6194011e20 con 0x7f61a0108740 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.840+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f61a019f250 con 0x7f61a0108740 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.843+0000 7f619dffb700 1 -- 172.21.15.107:0/1476455890 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f6194023390 con 0x7f61a0108740 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 msgr2=0x7f619003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 0x7f619003b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c006e40 comp rx=0 tx=0).stop 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 msgr2=0x7f61a019bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f61940116a0 tx=0x7f6194011ba0 comp rx=0 tx=0).stop 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 shutdown_connections 2024-04-24T23:35:57.382 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61900390f0 0x7f619003b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 --2- 172.21.15.107:0/1476455890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61a0108740 0x7f61a019bbc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 >> 172.21.15.107:0/1476455890 conn(0x7f61a0102170 msgr2=0x7f61a0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 shutdown_connections 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-04-24T23:35:56.845+0000 7f61a6bc3700 1 -- 172.21.15.107:0/1476455890 wait complete. 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout:Or, if you are only running a single cluster on this host: 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.383 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: ceph telemetry on 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.384 INFO:teuthology.orchestra.run.smithi107.stdout:For more information see: 2024-04-24T23:35:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-24T23:35:57.385 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:35:57.385 INFO:teuthology.orchestra.run.smithi107.stdout:Bootstrap complete. 2024-04-24T23:35:57.411 INFO:tasks.cephadm:Fetching config... 2024-04-24T23:35:57.411 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:35:57.412 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-24T23:35:57.427 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-24T23:35:57.427 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:35:57.427 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-24T23:35:57.481 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-24T23:35:57.482 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:35:57.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-24T23:35:57.545 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-24T23:35:57.546 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:35:57.546 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-24T23:35:57.604 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-24T23:35:57.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T23:35:57.675 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:57.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T23:35:57.727 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:57.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T23:35:57.778 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS7Qvfj7i2fotwQ6Qbfb1Nrrakk0sGWNJs2e3eX8C2t+XopxMRW0LCtB46nUkNo9i3dVc+yYtt5nJQUQBxn0sJEp5qK4G7fi1weVXiLjUzt1xz64TRIg9N4uR0/gPfVqLy6dOMs4OnNcD0+1xNEVgoPlegp3WxKRkkCEPBoOwNOLVS4BSb6FUTaoxIbjz7OOogiMeF3GNClVFJN9JJGuxYDGx0AdC8POUJdgKwUNljWPmquEf8ITmJPpN0FDJLXawHOOokkJhhnTD40WzX5lmNIXW6FpHBAmYioPiS5xeVzn+hdshddGYyZaszHmHAHAMXaj51yTK4CNaJLCRoOOvaLzZM8bNAvzoeDp/Qi3gis73uxLcz/1onEYPabe7gJJjudYRDaSBMKNlpVEhLaKUrb4zddbsEv6mpue8n+xeqt2fTa8f7pFDo7ujSVCLGp0qJLCW0DwAOqMBqUwsEgDktIWEHsMxXf//YGSzYJWHv+Ta/jHg8on6hiqcJvLOpb8= ceph-16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:35:57.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-24T23:35:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:35:58.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:35:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1476455890' entity='client.admin' 2024-04-24T23:36:00.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.265+0000 7f4e116a6700 1 -- 172.21.15.107:0/206470216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 msgr2=0x7f4e0c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:00.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.265+0000 7f4e116a6700 1 --2- 172.21.15.107:0/206470216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1020a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400b0a0 tx=0x7f4e0400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:36:00.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- 172.21.15.107:0/206470216 shutdown_connections 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 --2- 172.21.15.107:0/206470216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1020a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- 172.21.15.107:0/206470216 >> 172.21.15.107:0/206470216 conn(0x7f4e0c0fd220 msgr2=0x7f4e0c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- 172.21.15.107:0/206470216 shutdown_connections 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- 172.21.15.107:0/206470216 wait complete. 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 Processor -- start 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- start start 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.266+0000 7f4e116a6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c107d10 con 0x7f4e0c101c80 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e0affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e0affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53030/0 (socket says 172.21.15.107:53030) 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e0affd700 1 -- 172.21.15.107:0/114775762 learned_addr learned my addr 172.21.15.107:0/114775762 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e0affd700 1 -- 172.21.15.107:0/114775762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c199a30 con 0x7f4e0c101c80 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e0affd700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e04007890 tx=0x7f4e0400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e04017040 con 0x7f4e0c101c80 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e04022070 con 0x7f4e0c101c80 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c199cc0 con 0x7f4e0c101c80 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e0401d5b0 con 0x7f4e0c101c80 2024-04-24T23:36:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.267+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c19c9b0 con 0x7f4e0c101c80 2024-04-24T23:36:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.268+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f4e0401d8b0 con 0x7f4e0c101c80 2024-04-24T23:36:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.268+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0c108490 con 0x7f4e0c101c80 2024-04-24T23:36:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.268+0000 7f4df3fff700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 0x7f4df403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.268+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e0400f050 con 0x7f4e0c101c80 2024-04-24T23:36:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.268+0000 7f4e0a7fc700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 0x7f4df403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:00.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.269+0000 7f4e0a7fc700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 0x7f4df403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e00006fd0 tx=0x7f4e00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:00.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.273+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e04007e20 con 0x7f4e0c101c80 2024-04-24T23:36:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.466+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4e0c02dc40 con 0x7f4e0c101c80 2024-04-24T23:36:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.468+0000 7f4df3fff700 1 -- 172.21.15.107:0/114775762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4e04025c00 con 0x7f4e0c101c80 2024-04-24T23:36:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 msgr2=0x7f4df403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 0x7f4df403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e00006fd0 tx=0x7f4e00006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 msgr2=0x7f4e0c1994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e04007890 tx=0x7f4e0400bd40 comp rx=0 tx=0).stop 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 shutdown_connections 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4df40390f0 0x7f4df403b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 --2- 172.21.15.107:0/114775762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e0c101c80 0x7f4e0c1994f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 >> 172.21.15.107:0/114775762 conn(0x7f4e0c0fd220 msgr2=0x7f4e0c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 shutdown_connections 2024-04-24T23:36:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:00.471+0000 7f4e116a6700 1 -- 172.21.15.107:0/114775762 wait complete. 2024-04-24T23:36:01.083 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/114775762' entity='client.admin' 2024-04-24T23:36:01.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:01.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:01.109 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-24T23:36:01.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-24T23:36:01.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- 172.21.15.107:0/3097942951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 msgr2=0x7ff25c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3097942951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c1020a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff258009910 tx=0x7ff258009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- 172.21.15.107:0/3097942951 shutdown_connections 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3097942951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c1020a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- 172.21.15.107:0/3097942951 >> 172.21.15.107:0/3097942951 conn(0x7ff25c0fd220 msgr2=0x7ff25c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- 172.21.15.107:0/3097942951 shutdown_connections 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- 172.21.15.107:0/3097942951 wait complete. 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 Processor -- start 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.786+0000 7ff2637c2700 1 -- start start 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2637c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2637c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c107d10 con 0x7ff25c101c80 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff26155e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff26155e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53046/0 (socket says 172.21.15.107:53046) 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff26155e700 1 -- 172.21.15.107:0/3312497563 learned_addr learned my addr 172.21.15.107:0/3312497563 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff26155e700 1 -- 172.21.15.107:0/3312497563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c197960 con 0x7ff25c101c80 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff26155e700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff258006010 tx=0x7ff258005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff258027070 con 0x7ff25c101c80 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c197bf0 con 0x7ff25c101c80 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25800be40 con 0x7ff25c101c80 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff25800f460 con 0x7ff25c101c80 2024-04-24T23:36:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.787+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c19a890 con 0x7ff25c101c80 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.788+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ff25802a070 con 0x7ff25c101c80 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.788+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff25c108490 con 0x7ff25c101c80 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.788+0000 7ff2527fc700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 0x7ff24803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.788+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff25801e030 con 0x7ff25c101c80 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.789+0000 7ff260d5d700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 0x7ff24803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.789+0000 7ff260d5d700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 0x7ff24803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff24c006fd0 tx=0x7ff24c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:02.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.793+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff258013070 con 0x7ff25c101c80 2024-04-24T23:36:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.995+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ff25c02dc40 con 0x7ff2480390f0 2024-04-24T23:36:02.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:02.999+0000 7ff2527fc700 1 -- 172.21.15.107:0/3312497563 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff25c02dc40 con 0x7ff2480390f0 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 msgr2=0x7ff24803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 0x7ff24803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff24c006fd0 tx=0x7ff24c006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 msgr2=0x7ff25c197420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff258006010 tx=0x7ff258005050 comp rx=0 tx=0).stop 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 shutdown_connections 2024-04-24T23:36:03.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2480390f0 0x7ff24803b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 --2- 172.21.15.107:0/3312497563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff25c101c80 0x7ff25c197420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 >> 172.21.15.107:0/3312497563 conn(0x7ff25c0fd220 msgr2=0x7ff25c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 shutdown_connections 2024-04-24T23:36:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:03.001+0000 7ff2637c2700 1 -- 172.21.15.107:0/3312497563 wait complete. 2024-04-24T23:36:03.768 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-04-24T23:36:03.768 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:36:03.768 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-04-24T23:36:03.788 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:36:03.788 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:03.845 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-04-24T23:36:03.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch host add smithi170 2024-04-24T23:36:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:04.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: Updating smithi107:/etc/ceph/ceph.conf 2024-04-24T23:36:04.272 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:04.272 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:04.272 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:05.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:04 smithi107 ceph-mon[29118]: Updating smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:05.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:04 smithi107 ceph-mon[29118]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:05.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:04 smithi107 ceph-mon[29118]: Updating smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 -- 172.21.15.107:0/3098773549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 msgr2=0x7fed1c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3098773549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c100860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b0a0 tx=0x7fed0c00b3b0 comp rx=0 tx=0).stop 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 -- 172.21.15.107:0/3098773549 shutdown_connections 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3098773549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c100860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 -- 172.21.15.107:0/3098773549 >> 172.21.15.107:0/3098773549 conn(0x7fed1c077220 msgr2=0x7fed1c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 -- 172.21.15.107:0/3098773549 shutdown_connections 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 -- 172.21.15.107:0/3098773549 wait complete. 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.548+0000 7fed21e0a700 1 Processor -- start 2024-04-24T23:36:05.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed21e0a700 1 -- start start 2024-04-24T23:36:05.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed21e0a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:05.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed21e0a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c10c260 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed1b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed1b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53066/0 (socket says 172.21.15.107:53066) 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed1b7fe700 1 -- 172.21.15.107:0/3414355205 learned_addr learned my addr 172.21.15.107:0/3414355205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed1b7fe700 1 -- 172.21.15.107:0/3414355205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c197c60 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.549+0000 7fed1b7fe700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed0c015040 tx=0x7fed0c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.550+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed0c017040 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.550+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c197ef0 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.550+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c022070 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.550+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed0c01d5d0 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.550+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c19bfa0 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fed0c01d8d0 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c103540 con 0x7fed1c100480 2024-04-24T23:36:05.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed18ff9700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 0x7fed0403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:05.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fed0c00f050 con 0x7fed1c100480 2024-04-24T23:36:05.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed1affd700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 0x7fed0403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:05.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.551+0000 7fed1affd700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 0x7fed0403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed10006fd0 tx=0x7fed10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:05.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.556+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed0c007e20 con 0x7fed1c100480 2024-04-24T23:36:05.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:05.756+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}) -- 0x7fed1c198a30 con 0x7fed040390f0 2024-04-24T23:36:06.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:06 smithi107 ceph-mon[29118]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:07 smithi107 ceph-mon[29118]: Deploying cephadm binary to smithi170 2024-04-24T23:36:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:08 smithi107 ceph-mon[29118]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:09.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.837+0000 7fed18ff9700 1 -- 172.21.15.107:0/3414355205 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fed1c198a30 con 0x7fed040390f0 2024-04-24T23:36:09.838 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-04-24T23:36:09.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 msgr2=0x7fed0403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 0x7fed0403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed10006fd0 tx=0x7fed10006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 msgr2=0x7fed1c197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed0c015040 tx=0x7fed0c003d50 comp rx=0 tx=0).stop 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 shutdown_connections 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed040390f0 0x7fed0403b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 --2- 172.21.15.107:0/3414355205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed1c100480 0x7fed1c197720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 >> 172.21.15.107:0/3414355205 conn(0x7fed1c077220 msgr2=0x7fed1c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.839+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 shutdown_connections 2024-04-24T23:36:09.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:09.840+0000 7fed21e0a700 1 -- 172.21.15.107:0/3414355205 wait complete. 2024-04-24T23:36:10.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T23:36:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:11.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:11.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:10 smithi107 ceph-mon[29118]: Added host smithi170 2024-04-24T23:36:11.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:11.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:10 smithi107 ceph-mon[29118]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:11.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:12.075 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.076+0000 7fcdd5579700 1 -- 172.21.15.107:0/316534577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 msgr2=0x7fcdd01022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.076+0000 7fcdd5579700 1 --2- 172.21.15.107:0/316534577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd01022a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009910 tx=0x7fcdc0009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 -- 172.21.15.107:0/316534577 shutdown_connections 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 --2- 172.21.15.107:0/316534577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd01022a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 -- 172.21.15.107:0/316534577 >> 172.21.15.107:0/316534577 conn(0x7fcdd00fd750 msgr2=0x7fcdd00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:12.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 -- 172.21.15.107:0/316534577 shutdown_connections 2024-04-24T23:36:12.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 -- 172.21.15.107:0/316534577 wait complete. 2024-04-24T23:36:12.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 Processor -- start 2024-04-24T23:36:12.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.077+0000 7fcdd5579700 1 -- start start 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdd5579700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdd5579700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010c290 con 0x7fcdd0101ec0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdceffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdceffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51584/0 (socket says 172.21.15.107:51584) 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdceffd700 1 -- 172.21.15.107:0/514334490 learned_addr learned my addr 172.21.15.107:0/514334490 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdceffd700 1 -- 172.21.15.107:0/514334490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd019d940 con 0x7fcdd0101ec0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdceffd700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005850 tx=0x7fcdc0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc0026070 con 0x7fcdd0101ec0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc000bbd0 con 0x7fcdd0101ec0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd019dc30 con 0x7fcdd0101ec0 2024-04-24T23:36:12.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc0003970 con 0x7fcdd0101ec0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.078+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd01975c0 con 0x7fcdd0101ec0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.079+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fcdc0029070 con 0x7fcdd0101ec0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.079+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd01033a0 con 0x7fcdd0101ec0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.079+0000 7fcdb7fff700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 0x7fcdb803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.079+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcdc001e070 con 0x7fcdd0101ec0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.079+0000 7fcdce7fc700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 0x7fcdb803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.080+0000 7fcdce7fc700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 0x7fcdb803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4006fd0 tx=0x7fcdc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:12.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.084+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdc0013070 con 0x7fcdd0101ec0 2024-04-24T23:36:12.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.283+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcdd0198af0 con 0x7fcdb80390f0 2024-04-24T23:36:12.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.285+0000 7fcdb7fff700 1 -- 172.21.15.107:0/514334490 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fcdd0198af0 con 0x7fcdb80390f0 2024-04-24T23:36:12.285 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:36:12.285 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 msgr2=0x7fcdb803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 0x7fcdb803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4006fd0 tx=0x7fcdc4006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 msgr2=0x7fcdd019d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005850 tx=0x7fcdc0005030 comp rx=0 tx=0).stop 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 shutdown_connections 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcdb80390f0 0x7fcdb803b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 --2- 172.21.15.107:0/514334490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdd0101ec0 0x7fcdd019d400 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:12.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 >> 172.21.15.107:0/514334490 conn(0x7fcdd00fd750 msgr2=0x7fcdd0103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:12.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 shutdown_connections 2024-04-24T23:36:12.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:12.287+0000 7fcdd5579700 1 -- 172.21.15.107:0/514334490 wait complete. 2024-04-24T23:36:12.878 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-04-24T23:36:12.879 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:36:12.879 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-04-24T23:36:12.898 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:36:12.898 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:12.955 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-04-24T23:36:12.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch host add smithi195 2024-04-24T23:36:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:13.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:12 smithi107 ceph-mon[29118]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:13.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:14.209 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:13 smithi107 ceph-mon[29118]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/319244722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 msgr2=0x7f8d98100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/319244722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d98100860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009910 tx=0x7f8d88009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/319244722 shutdown_connections 2024-04-24T23:36:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/319244722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d98100860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/319244722 >> 172.21.15.107:0/319244722 conn(0x7f8d98077220 msgr2=0x7f8d98077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/319244722 shutdown_connections 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/319244722 wait complete. 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.662+0000 7f8d9d3ac700 1 Processor -- start 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d9d3ac700 1 -- start start 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d9d3ac700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d9d3ac700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9810c230 con 0x7f8d98100480 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d96ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d96ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51592/0 (socket says 172.21.15.107:51592) 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d96ffd700 1 -- 172.21.15.107:0/1105230752 learned_addr learned my addr 172.21.15.107:0/1105230752 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:14.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d96ffd700 1 -- 172.21.15.107:0/1105230752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9819d950 con 0x7f8d98100480 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.663+0000 7f8d96ffd700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d88005850 tx=0x7f8d88005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d88026070 con 0x7f8d98100480 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8800bbd0 con 0x7f8d98100480 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9819dc40 con 0x7f8d98100480 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d88003970 con 0x7f8d98100480 2024-04-24T23:36:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d981975d0 con 0x7f8d98100480 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.664+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f8d88029070 con 0x7f8d98100480 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.665+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d98103540 con 0x7f8d98100480 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.665+0000 7f8d94ff9700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 0x7f8d8403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.665+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8d8801e070 con 0x7f8d98100480 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.665+0000 7f8d8e7fc700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 0x7f8d8403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.665+0000 7f8d8e7fc700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 0x7f8d8403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d80006fd0 tx=0x7f8d80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:14.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.670+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d88013070 con 0x7f8d98100480 2024-04-24T23:36:14.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:14.869+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}) -- 0x7f8d98063890 con 0x7f8d840390f0 2024-04-24T23:36:15.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:14 smithi107 ceph-mon[29118]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:16 smithi107 ceph-mon[29118]: Deploying cephadm binary to smithi195 2024-04-24T23:36:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:16 smithi107 ceph-mon[29118]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:16 smithi107 ceph-mon[29118]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:16 smithi107 ceph-mon[29118]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:16 smithi107 ceph-mon[29118]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:18.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.933+0000 7f8d94ff9700 1 -- 172.21.15.107:0/1105230752 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8d98063890 con 0x7f8d840390f0 2024-04-24T23:36:18.934 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 msgr2=0x7f8d8403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 0x7f8d8403b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d80006fd0 tx=0x7f8d80006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 msgr2=0x7f8d9819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d88005850 tx=0x7f8d88005030 comp rx=0 tx=0).stop 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 shutdown_connections 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d840390f0 0x7f8d8403b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 --2- 172.21.15.107:0/1105230752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d98100480 0x7f8d9819d410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 >> 172.21.15.107:0/1105230752 conn(0x7f8d98077220 msgr2=0x7f8d98103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 shutdown_connections 2024-04-24T23:36:18.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:18.936+0000 7f8d9d3ac700 1 -- 172.21.15.107:0/1105230752 wait complete. 2024-04-24T23:36:19.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:18 smithi107 ceph-mon[29118]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:19.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:19.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:19.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:19.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T23:36:19.704 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:20.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:19 smithi107 ceph-mon[29118]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:20.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:20.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:19 smithi107 ceph-mon[29118]: Added host smithi195 2024-04-24T23:36:20.118 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:20.118 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:21.005 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:20 smithi107 ceph-mon[29118]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:21.005 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 -- 172.21.15.107:0/860374992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 msgr2=0x7f3c04101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 --2- 172.21.15.107:0/860374992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04101180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009910 tx=0x7f3bf8009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 -- 172.21.15.107:0/860374992 shutdown_connections 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 --2- 172.21.15.107:0/860374992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04101180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 -- 172.21.15.107:0/860374992 >> 172.21.15.107:0/860374992 conn(0x7f3c040fc670 msgr2=0x7f3c040fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:21.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 -- 172.21.15.107:0/860374992 shutdown_connections 2024-04-24T23:36:21.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.208+0000 7f3c0b256700 1 -- 172.21.15.107:0/860374992 wait complete. 2024-04-24T23:36:21.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c0b256700 1 Processor -- start 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c0b256700 1 -- start start 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c0b256700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c0b256700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0410b170 con 0x7f3c04100da0 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c08ff2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c08ff2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59800/0 (socket says 172.21.15.107:59800) 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c08ff2700 1 -- 172.21.15.107:0/2788904424 learned_addr learned my addr 172.21.15.107:0/2788904424 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c08ff2700 1 -- 172.21.15.107:0/2788904424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c04198b40 con 0x7f3c04100da0 2024-04-24T23:36:21.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c08ff2700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005850 tx=0x7f3bf8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:21.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf8026070 con 0x7f3c04100da0 2024-04-24T23:36:21.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bf800bbd0 con 0x7f3c04100da0 2024-04-24T23:36:21.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.209+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c04198e30 con 0x7f3c04100da0 2024-04-24T23:36:21.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.210+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf8003970 con 0x7f3c04100da0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.210+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c041992f0 con 0x7f3c04100da0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.210+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f3bf8029070 con 0x7f3c04100da0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.211+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c04102280 con 0x7f3c04100da0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.211+0000 7f3c01ffb700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 0x7f3bf403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.211+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3bf801e070 con 0x7f3c04100da0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.211+0000 7f3c03fff700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 0x7f3bf403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:21.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.211+0000 7f3c03fff700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 0x7f3bf403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:21.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.216+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3bf8013070 con 0x7f3c04100da0 2024-04-24T23:36:21.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.418+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3c0406a610 con 0x7f3bf4039150 2024-04-24T23:36:21.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.419+0000 7f3c01ffb700 1 -- 172.21.15.107:0/2788904424 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f3c0406a610 con 0x7f3bf4039150 2024-04-24T23:36:21.419 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:36:21.420 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 msgr2=0x7f3bf403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 0x7f3bf403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 msgr2=0x7f3c04198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005850 tx=0x7f3bf8005030 comp rx=0 tx=0).stop 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 shutdown_connections 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3bf4039150 0x7f3bf403b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 --2- 172.21.15.107:0/2788904424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04100da0 0x7f3c04198600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:21.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 >> 172.21.15.107:0/2788904424 conn(0x7f3c040fc670 msgr2=0x7f3c04102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:21.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 shutdown_connections 2024-04-24T23:36:21.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:21.421+0000 7f3c0b256700 1 -- 172.21.15.107:0/2788904424 wait complete. 2024-04-24T23:36:22.104 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-24T23:36:22.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-24T23:36:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:22.655 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:22 smithi107 ceph-mon[29118]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:22.655 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:22.655 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:22 smithi107 ceph-mon[29118]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 -- 172.21.15.107:0/3813488811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 msgr2=0x7f97b4102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 --2- 172.21.15.107:0/3813488811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b4102260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009910 tx=0x7f97a8009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 -- 172.21.15.107:0/3813488811 shutdown_connections 2024-04-24T23:36:23.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 --2- 172.21.15.107:0/3813488811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b4102260 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 -- 172.21.15.107:0/3813488811 >> 172.21.15.107:0/3813488811 conn(0x7f97b40fd750 msgr2=0x7f97b40ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 -- 172.21.15.107:0/3813488811 shutdown_connections 2024-04-24T23:36:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.653+0000 7f97ba49d700 1 -- 172.21.15.107:0/3813488811 wait complete. 2024-04-24T23:36:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97ba49d700 1 Processor -- start 2024-04-24T23:36:23.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97ba49d700 1 -- start start 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97ba49d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97ba49d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410c250 con 0x7f97b4101e80 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97b3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59818/0 (socket says 172.21.15.107:59818) 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.654+0000 7f97b3fff700 1 -- 172.21.15.107:0/301477083 learned_addr learned my addr 172.21.15.107:0/301477083 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:23.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97b3fff700 1 -- 172.21.15.107:0/301477083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b419d910 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97b3fff700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97a8005850 tx=0x7f97a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97a8026070 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b419dc00 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a800bbd0 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97a8003970 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.655+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b4197590 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f97a8029070 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b4103360 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97b17fa700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 0x7f979c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f97a801e070 con 0x7f97b4101e80 2024-04-24T23:36:23.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97b37fe700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 0x7f979c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:23.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.656+0000 7f97b37fe700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 0x7f979c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f97a4006fd0 tx=0x7f97a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:23.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.661+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f97a8013070 con 0x7f97b4101e80 2024-04-24T23:36:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.862+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f97b4063890 con 0x7f97b4101e80 2024-04-24T23:36:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.913+0000 7f97b17fa700 1 -- 172.21.15.107:0/301477083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f97a8030020 con 0x7f97b4101e80 2024-04-24T23:36:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 msgr2=0x7f979c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 0x7f979c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f97a4006fd0 tx=0x7f97a4006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 msgr2=0x7f97b419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97a8005850 tx=0x7f97a8005030 comp rx=0 tx=0).stop 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 shutdown_connections 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f979c0390f0 0x7f979c03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 --2- 172.21.15.107:0/301477083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e80 0x7f97b419d3d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 >> 172.21.15.107:0/301477083 conn(0x7f97b40fd750 msgr2=0x7f97b4103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 shutdown_connections 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:23.915+0000 7f97ba49d700 1 -- 172.21.15.107:0/301477083 wait complete. 2024-04-24T23:36:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-04-24T23:36:24.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:23 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T23:36:24.690 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-04-24T23:36:24.691 INFO:tasks.cephadm:Adding mon.b on smithi170 2024-04-24T23:36:24.691 INFO:tasks.cephadm:Adding mon.c on smithi195 2024-04-24T23:36:24.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c' 2024-04-24T23:36:24.920 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-24T23:36:24.967 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-24T23:36:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T23:36:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:25.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:25.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:25.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:25.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:25.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:25.380 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:24 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 -- 172.21.15.195:0/2709839371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 msgr2=0x7f268c106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 --2- 172.21.15.195:0/2709839371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c106fd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f2688009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 -- 172.21.15.195:0/2709839371 shutdown_connections 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 --2- 172.21.15.195:0/2709839371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c106fd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 -- 172.21.15.195:0/2709839371 >> 172.21.15.195:0/2709839371 conn(0x7f268c0fc630 msgr2=0x7f268c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 -- 172.21.15.195:0/2709839371 shutdown_connections 2024-04-24T23:36:26.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.435+0000 7f2693b8a700 1 -- 172.21.15.195:0/2709839371 wait complete. 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2693b8a700 1 Processor -- start 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2693b8a700 1 -- start start 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2693b8a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2693b8a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10b1b0 con 0x7f268c106bf0 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2691926700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:26.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2691926700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.195:38664/0 (socket says 172.21.15.195:38664) 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.436+0000 7f2691926700 1 -- 172.21.15.195:0/3671300768 learned_addr learned my addr 172.21.15.195:0/3671300768 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2691926700 1 -- 172.21.15.195:0/3671300768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c196b30 con 0x7f268c106bf0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2691926700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2688005850 tx=0x7f2688005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2688026070 con 0x7f268c106bf0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c196dc0 con 0x7f268c106bf0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268800bc30 con 0x7f268c106bf0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2688003a20 con 0x7f268c106bf0 2024-04-24T23:36:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.437+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c19aec0 con 0x7f268c106bf0 2024-04-24T23:36:26.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.438+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f2688029070 con 0x7f268c106bf0 2024-04-24T23:36:26.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.438+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268c1021f0 con 0x7f268c106bf0 2024-04-24T23:36:26.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.438+0000 7f2682ffd700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 0x7f267803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:26.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.438+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f268801e070 con 0x7f268c106bf0 2024-04-24T23:36:26.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.438+0000 7f2691125700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 0x7f267803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:26.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.439+0000 7f2691125700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 0x7f267803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f267c006fd0 tx=0x7f267c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:26.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.444+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2688013070 con 0x7f268c106bf0 2024-04-24T23:36:26.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.650+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}) -- 0x7f268c197950 con 0x7f26780390f0 2024-04-24T23:36:26.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.657+0000 7f2682ffd700 1 -- 172.21.15.195:0/3671300768 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f268c197950 con 0x7f26780390f0 2024-04-24T23:36:26.658 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 msgr2=0x7f267803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 0x7f267803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f267c006fd0 tx=0x7f267c006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 msgr2=0x7f268c1965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2688005850 tx=0x7f2688005030 comp rx=0 tx=0).stop 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 shutdown_connections 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26780390f0 0x7f267803b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 --2- 172.21.15.195:0/3671300768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f268c106bf0 0x7f268c1965f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.659+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 >> 172.21.15.195:0/3671300768 conn(0x7f268c0fc630 msgr2=0x7f268c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.660+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 shutdown_connections 2024-04-24T23:36:26.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:26.660+0000 7f2693b8a700 1 -- 172.21.15.195:0/3671300768 wait complete. 2024-04-24T23:36:27.233 DEBUG:teuthology.orchestra.run.smithi170:mon.b> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.b.service 2024-04-24T23:36:27.236 DEBUG:teuthology.orchestra.run.smithi195:mon.c> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.c.service 2024-04-24T23:36:27.239 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T23:36:27.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:27.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:28.331 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:28.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:27 smithi107 ceph-mon[29118]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:28.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:27 smithi107 ceph-mon[29118]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c;count:3 2024-04-24T23:36:28.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:27 smithi107 ceph-mon[29118]: Deploying daemon mon.c on smithi195 2024-04-24T23:36:29.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:28 smithi107 ceph-mon[29118]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:30.495 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:30 smithi195 systemd[1]: Starting Ceph mon.c for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:36:30.983 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- 172.21.15.195:0/3491905124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 msgr2=0x7f867c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 --2- 172.21.15.195:0/3491905124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c100830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8664009910 tx=0x7f8664009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- 172.21.15.195:0/3491905124 shutdown_connections 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 --2- 172.21.15.195:0/3491905124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c100830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- 172.21.15.195:0/3491905124 >> 172.21.15.195:0/3491905124 conn(0x7f867c0771e0 msgr2=0x7f867c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- 172.21.15.195:0/3491905124 shutdown_connections 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- 172.21.15.195:0/3491905124 wait complete. 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 Processor -- start 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.982+0000 7f8682826700 1 -- start start 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f8682826700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f8682826700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f867c10c2e0 con 0x7f867c100450 2024-04-24T23:36:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f867bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:30.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f867bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.195:38690/0 (socket says 172.21.15.195:38690) 2024-04-24T23:36:30.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f867bfff700 1 -- 172.21.15.195:0/2587336194 learned_addr learned my addr 172.21.15.195:0/2587336194 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.983+0000 7f867bfff700 1 -- 172.21.15.195:0/2587336194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c19d950 con 0x7f867c100450 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f867bfff700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8664005850 tx=0x7f8664005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8664026070 con 0x7f867c100450 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f867c19dc40 con 0x7f867c100450 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f866400bbd0 con 0x7f867c100450 2024-04-24T23:36:30.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8664003970 con 0x7f867c100450 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.984+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c1975d0 con 0x7f867c100450 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.985+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f8664029070 con 0x7f867c100450 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.985+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f867c103510 con 0x7f867c100450 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.985+0000 7f86797fa700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 0x7f866803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.985+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f866401e070 con 0x7f867c100450 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.985+0000 7f867b7fe700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 0x7f866803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:30.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.986+0000 7f867b7fe700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 0x7f866803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f866c006fd0 tx=0x7f866c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:30.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:30.990+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8664013070 con 0x7f867c100450 2024-04-24T23:36:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.280+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f867c198b00 con 0x7f867c100450 2024-04-24T23:36:31.281 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.281+0000 7f86797fa700 1 -- 172.21.15.195:0/2587336194 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f8664030020 con 0x7f867c100450 2024-04-24T23:36:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:36:31.282 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","modified":"2024-04-24T23:34:42.177629Z","created":"2024-04-24T23:34:42.177629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T23:36:31.283 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 msgr2=0x7f866803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 0x7f866803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f866c006fd0 tx=0x7f866c006e40 comp rx=0 tx=0).stop 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 msgr2=0x7f867c19d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8664005850 tx=0x7f8664005030 comp rx=0 tx=0).stop 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 shutdown_connections 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86680390f0 0x7f866803b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 --2- 172.21.15.195:0/2587336194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f867c100450 0x7f867c19d410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 >> 172.21.15.195:0/2587336194 conn(0x7f867c0771e0 msgr2=0x7f867c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 shutdown_connections 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:31.283+0000 7f8682826700 1 -- 172.21.15.195:0/2587336194 wait complete. 2024-04-24T23:36:31.284 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-04-24T23:36:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:30 smithi107 ceph-mon[29118]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.404 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 podman[32955]: 2024-04-24 23:36:31.044933792 +0000 UTC m=+0.467985692 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 podman[32955]: 2024-04-24 23:36:31.17426316 +0000 UTC m=+0.597315028 container create 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 podman[32955]: 2024-04-24 23:36:31.351912621 +0000 UTC m=+0.774964489 container init 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 podman[32955]: 2024-04-24 23:36:31.355935572 +0000 UTC m=+0.778987441 container start 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pidfile_write: ignore empty --pid-file 2024-04-24T23:36:31.405 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: load: jerasure load: lrc 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: RocksDB version: 7.9.2 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Git sha 0 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: DB SUMMARY 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: DB Session ID: IZREW78D1MT48RS70QLR 2024-04-24T23:36:31.406 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: CURRENT file: CURRENT 2024-04-24T23:36:31.407 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T23:36:31.407 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-24T23:36:31.407 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-24T23:36:31.407 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.error_if_exists: 0 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.create_if_missing: 0 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.paranoid_checks: 1 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T23:36:31.411 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.env: 0x55b8098c0c60 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.info_log: 0x55b80b00faa0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.statistics: (nil) 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.use_fsync: 0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_log_file_size: 0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_fallocate: 1 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T23:36:31.412 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.use_direct_reads: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.db_log_dir: 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.wal_dir: 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T23:36:31.413 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.write_buffer_manager: 0x55b80afe21e0 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T23:36:31.414 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.unordered_write: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.row_cache: None 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.wal_filter: None 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.two_write_queues: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.wal_compression: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.atomic_flush: 0 2024-04-24T23:36:31.415 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.log_readahead_size: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_background_jobs: 2 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_background_compactions: -1 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_subcompactions: 1 2024-04-24T23:36:31.416 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_open_files: -1 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_background_flushes: -1 2024-04-24T23:36:31.417 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Compression algorithms supported: 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kZSTD supported: 0 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kXpressCompression supported: 0 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kZlibCompression supported: 1 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kSnappyCompression supported: 1 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kLZ4Compression supported: 1 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: kBZip2Compression supported: 0 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T23:36:31.418 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.merge_operator: 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_filter: None 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b80b00fba0) 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: cache_index_and_filter_blocks: 1 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: pin_top_level_index_and_filter: 1 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: index_type: 0 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: data_block_index_type: 0 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: index_shortening: 1 2024-04-24T23:36:31.419 INFO:journalctl@ceph.mon.c.smithi195.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: checksum: 4 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: no_block_cache: 0 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_cache: 0x55b80b013090 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_cache_name: BinnedLRUCache 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_cache_options: 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: capacity : 536870912 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: num_shard_bits : 4 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: strict_capacity_limit : 0 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: high_pri_pool_ratio: 0.000 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_cache_compressed: (nil) 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: persistent_cache: (nil) 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_size: 4096 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_size_deviation: 10 2024-04-24T23:36:31.420 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_restart_interval: 16 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: index_block_restart_interval: 1 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: metadata_block_size: 4096 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: partition_filters: 0 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: use_delta_encoding: 1 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: filter_policy: bloomfilter 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: whole_key_filtering: 1 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: verify_compression: 0 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: read_amp_bytes_per_bit: 0 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: format_version: 5 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: enable_index_compression: 1 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: block_align: 0 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: max_auto_readahead_size: 262144 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: prepopulate_block_cache: 0 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: initial_auto_readahead_size: 8192 2024-04-24T23:36:31.421 INFO:journalctl@ceph.mon.c.smithi195.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression: NoCompression 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.num_levels: 7 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T23:36:31.422 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T23:36:31.423 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T23:36:31.424 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T23:36:31.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.table_properties_collectors: 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.inplace_update_support: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.bloom_locality: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.max_successive_merges: 0 2024-04-24T23:36:31.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.ttl: 2592000 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enable_blob_files: false 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.min_blob_size: 0 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T23:36:31.427 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 42aa5d52-4a6a-473a-9f66-0a8a7fa0ffaa 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001791385754, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001791386302, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714001791, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "42aa5d52-4a6a-473a-9f66-0a8a7fa0ffaa", "db_session_id": "IZREW78D1MT48RS70QLR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001791386399, "job": 1, "event": "recovery_finished"} 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b80b104000 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: DB pointer 0x55b80b0ee000 2024-04-24T23:36:31.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** DB Stats ** 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** Compaction Stats [default] ** 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T23:36:31.429 INFO:journalctl@ceph.mon.c.smithi195.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** Compaction Stats [default] ** 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T23:36:31.430 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: Block cache BinnedLRUCache@0x55b80b013090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: using public_addr v2:172.21.15.195:0/0 -> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: starting mon.c rank -1 at public addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] at bind addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(???) e0 preinit fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).mds e1 new map 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-24T23:36:31.431 INFO:journalctl@ceph.mon.c.smithi195.stdout: e1 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout: legacy client fscid: -1 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout: No filesystems configured 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:31.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:31.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Deploying cephadm binary to smithi195 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Added host smithi195 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:31.434 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:31.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c;count:3 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: Deploying daemon mon.c on smithi195 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:31.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 ceph-mon[33019]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T23:36:31.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 bash[32955]: 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c 2024-04-24T23:36:31.766 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:31 smithi195 systemd[1]: Started Ceph mon.c for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:36:32.976 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T23:36:32.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T23:36:33.220 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:36:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: mon.a calling monitor election 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: mon.c calling monitor election 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: monmap epoch 2 2024-04-24T23:36:36.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: last_changed 2024-04-24T23:36:31.412396+0000 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: min_mon_release 19 (squid) 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: election_strategy: 1 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: fsmap 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: mgrmap e13: x(active, since 48s) 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: overall HEALTH_OK 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:36.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:36.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:36.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: mon.a calling monitor election 2024-04-24T23:36:36.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: mon.c calling monitor election 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: monmap epoch 2 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: last_changed 2024-04-24T23:36:31.412396+0000 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:36.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: min_mon_release 19 (squid) 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: election_strategy: 1 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: fsmap 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: mgrmap e13: x(active, since 48s) 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: overall HEALTH_OK 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:36.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 -- 172.21.15.195:0/1217291865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6400035c0 msgr2=0x7fb640005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 --2- 172.21.15.195:0/1217291865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6400035c0 0x7fb640005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb658100b70 tx=0x7fb650009e80 comp rx=0 tx=0).stop 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 -- 172.21.15.195:0/1217291865 shutdown_connections 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 --2- 172.21.15.195:0/1217291865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6400035c0 0x7fb640005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 -- 172.21.15.195:0/1217291865 >> 172.21.15.195:0/1217291865 conn(0x7fb6580fc770 msgr2=0x7fb6580feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 -- 172.21.15.195:0/1217291865 shutdown_connections 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.627+0000 7fb65f114700 1 -- 172.21.15.195:0/1217291865 wait complete. 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.628+0000 7fb65f114700 1 Processor -- start 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.628+0000 7fb65f114700 1 -- start start 2024-04-24T23:36:37.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.628+0000 7fb65f114700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65f114700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 0x7fb6581976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65f114700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb658108cd0 con 0x7fb65819db10 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65f114700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb658108b50 con 0x7fb6400035c0 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:46946/0 (socket says 172.21.15.195:46946) 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 -- 172.21.15.195:0/3274135313 learned_addr learned my addr 172.21.15.195:0/3274135313 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 -- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 msgr2=0x7fb65819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 -- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 msgr2=0x7fb65819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb65ceb0700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb657fff700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 0x7fb6581976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb657fff700 1 -- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 msgr2=0x7fb65819d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:36:37.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb657fff700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.629+0000 7fb657fff700 1 -- 172.21.15.195:0/3274135313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65819e090 con 0x7fb65819db10 2024-04-24T23:36:37.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb657fff700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 0x7fb6581976a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b4d0 tx=0x7fb64c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:37.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb64c011630 con 0x7fb65819db10 2024-04-24T23:36:37.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb658197c40 con 0x7fb65819db10 2024-04-24T23:36:37.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb64c011cb0 con 0x7fb65819db10 2024-04-24T23:36:37.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb64c010e60 con 0x7fb65819db10 2024-04-24T23:36:37.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.630+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6581981d0 con 0x7fb65819db10 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.631+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb64c018020 con 0x7fb65819db10 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.631+0000 7fb655ffb700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 0x7fb64403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.631+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb64c04bba0 con 0x7fb65819db10 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.631+0000 7fb65ceb0700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 0x7fb64403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.632+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb658102380 con 0x7fb65819db10 2024-04-24T23:36:37.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.632+0000 7fb65ceb0700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 0x7fb64403b300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb650019000 tx=0x7fb6500074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:37.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.637+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb64c01f020 con 0x7fb65819db10 2024-04-24T23:36:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:37 smithi195 ceph-mon[33019]: Deploying daemon mon.b on smithi170 2024-04-24T23:36:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:37 smithi107 ceph-mon[29118]: Deploying daemon mon.b on smithi170 2024-04-24T23:36:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:37.916 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.916+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb658198540 con 0x7fb65819db10 2024-04-24T23:36:37.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.917+0000 7fb655ffb700 1 -- 172.21.15.195:0/3274135313 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fb64c018310 con 0x7fb65819db10 2024-04-24T23:36:37.917 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:36:37.917 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","modified":"2024-04-24T23:36:31.412396Z","created":"2024-04-24T23:34:42.177629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T23:36:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 msgr2=0x7fb64403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 0x7fb64403b300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb650019000 tx=0x7fb6500074e0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 msgr2=0x7fb6581976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 0x7fb6581976a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b4d0 tx=0x7fb64c00b890 comp rx=0 tx=0).stop 2024-04-24T23:36:37.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 shutdown_connections 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb644038e40 0x7fb64403b300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6400035c0 0x7fb65819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 --2- 172.21.15.195:0/3274135313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb65819db10 0x7fb6581976a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 >> 172.21.15.195:0/3274135313 conn(0x7fb6580fc770 msgr2=0x7fb65806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 shutdown_connections 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:37.919+0000 7fb65f114700 1 -- 172.21.15.195:0/3274135313 wait complete. 2024-04-24T23:36:37.920 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-04-24T23:36:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:38 smithi195 ceph-mon[33019]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:38 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/3274135313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:38.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:38 smithi107 ceph-mon[29118]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:38.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:38 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/3274135313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:39.508 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T23:36:39.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T23:36:39.747 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:36:40.308 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T23:36:40.308 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.merge_operator: 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_filter: None 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55adba357ba0) 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks: 1 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T23:36:40.309 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_top_level_index_and_filter: 1 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_type: 0 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_index_type: 0 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_shortening: 1 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: checksum: 4 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: no_block_cache: 0 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache: 0x55adba35b090 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_name: BinnedLRUCache 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_options: 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: capacity : 536870912 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_shard_bits : 4 2024-04-24T23:36:40.310 INFO:journalctl@ceph.mon.b.smithi170.stdout: strict_capacity_limit : 0 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: high_pri_pool_ratio: 0.000 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_compressed: (nil) 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: persistent_cache: (nil) 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size: 4096 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size_deviation: 10 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_restart_interval: 16 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_block_restart_interval: 1 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: metadata_block_size: 4096 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: partition_filters: 0 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_delta_encoding: 1 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: filter_policy: bloomfilter 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: whole_key_filtering: 1 2024-04-24T23:36:40.311 INFO:journalctl@ceph.mon.b.smithi170.stdout: verify_compression: 0 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: read_amp_bytes_per_bit: 0 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: format_version: 5 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_index_compression: 1 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_align: 0 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: max_auto_readahead_size: 262144 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: prepopulate_block_cache: 0 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: initial_auto_readahead_size: 8192 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression: NoCompression 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T23:36:40.312 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.num_levels: 7 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T23:36:40.313 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T23:36:40.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T23:36:40.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T23:36:40.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T23:36:40.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:36:40.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T23:36:40.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T23:36:40.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T23:36:40.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T23:36:40.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T23:36:40.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T23:36:40.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T23:36:40.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.table_properties_collectors: 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.inplace_update_support: 0 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.bloom_locality: 0 2024-04-24T23:36:40.591 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.max_successive_merges: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.ttl: 2592000 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.enable_blob_files: false 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.min_blob_size: 0 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T23:36:40.592 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f5131107-36ac-4676-baf2-b7b34af332ed 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001800307605, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001800308038, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714001800, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f5131107-36ac-4676-baf2-b7b34af332ed", "db_session_id": "EMSOQQ5FXQLOZDE2SPCO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714001800308119, "job": 1, "event": "recovery_finished"} 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55adba44c000 2024-04-24T23:36:40.593 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: DB pointer 0x55adba436000 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** DB Stats ** 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:36:40.594 INFO:journalctl@ceph.mon.b.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T23:36:40.595 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache BinnedLRUCache@0x55adba35b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: using public_addr v2:172.21.15.170:0/0 -> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: starting mon.b rank -1 at public addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] at bind addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(???) e0 preinit fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:40.596 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).mds e1 new map 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: e1 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: legacy client fscid: -1 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout: No filesystems configured 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T23:36:40.597 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.598 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Deploying cephadm binary to smithi195 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.599 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Added host smithi195 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/301477083' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:40.601 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c;count:3 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Deploying daemon mon.c on smithi195 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.a calling monitor election 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.c calling monitor election 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: monmap epoch 2 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: last_changed 2024-04-24T23:36:31.412396+0000 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: min_mon_release 19 (squid) 2024-04-24T23:36:40.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: election_strategy: 1 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: fsmap 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mgrmap e13: x(active, since 48s) 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: overall HEALTH_OK 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: Deploying daemon mon.b on smithi170 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:40.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/3274135313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:40.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 ceph-mon[32714]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T23:36:40.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 bash[32695]: 0dbe7c606142af67d0f5087421d0028459cd33692bcb475e3b378f6e0b5659be 2024-04-24T23:36:40.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:40 smithi170 systemd[1]: Started Ceph mon.b for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: mon.a calling monitor election 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: mon.c calling monitor election 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:45.653 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: monmap epoch 3 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: min_mon_release 19 (squid) 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: election_strategy: 1 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:45.654 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: fsmap 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: mgrmap e13: x(active, since 57s) 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: overall HEALTH_OK 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.655 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.657 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 -- 172.21.15.195:0/1922638234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df40035c0 msgr2=0x7f3df4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:45.657 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 --2- 172.21.15.195:0/1922638234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df40035c0 0x7f3df4005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c100b70 tx=0x7f3e0000fe60 comp rx=0 tx=0).stop 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 -- 172.21.15.195:0/1922638234 shutdown_connections 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 --2- 172.21.15.195:0/1922638234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df40035c0 0x7f3df4005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 -- 172.21.15.195:0/1922638234 >> 172.21.15.195:0/1922638234 conn(0x7f3e0c0fc730 msgr2=0x7f3e0c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 -- 172.21.15.195:0/1922638234 shutdown_connections 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 -- 172.21.15.195:0/1922638234 wait complete. 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.652+0000 7f3e1226d700 1 Processor -- start 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 -- start start 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 0x7f3e0c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c108c80 con 0x7f3e0c197d80 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c108b00 con 0x7f3e0c19dbb0 2024-04-24T23:36:45.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1226d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c108e00 con 0x7f3df40035c0 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:56672/0 (socket says 172.21.15.195:56672) 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 -- 172.21.15.195:0/3672210937 learned_addr learned my addr 172.21.15.195:0/3672210937 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0b7fe700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 msgr2=0x7f3e0c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 msgr2=0x7f3e0c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e0bfff700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.653+0000 7f3e1080a700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 0x7f3e0c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e0b7fe700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 msgr2=0x7f3e0c197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e0b7fe700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 msgr2=0x7f3e0c197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e0bfff700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e0b7fe700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e0b7fe700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:36:45.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 msgr2=0x7f3e0c19d670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 msgr2=0x7f3e0c197840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 -- 172.21.15.195:0/3672210937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0c1a44d0 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1080a700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 0x7f3e0c198230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bd50 tx=0x7f3e0400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04019070 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e0c1a4760 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e0c1a4c70 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0400cd10 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.654+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0401d770 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.655+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f3e0401c070 con 0x7f3e0c197d80 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.656+0000 7f3e097fa700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 0x7f3df803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:45.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.656+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e0c102380 con 0x7f3e0c197d80 2024-04-24T23:36:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.656+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3e0404cbd0 con 0x7f3e0c197d80 2024-04-24T23:36:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.656+0000 7f3e0bfff700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 0x7f3df803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.656+0000 7f3e0bfff700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 0x7f3df803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e00001eb0 tx=0x7f3e00005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.661+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e0400e0a0 con 0x7f3e0c197d80 2024-04-24T23:36:45.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:45.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: mon.a calling monitor election 2024-04-24T23:36:45.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: mon.c calling monitor election 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: monmap epoch 3 2024-04-24T23:36:45.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: min_mon_release 19 (squid) 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: election_strategy: 1 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: fsmap 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: mgrmap e13: x(active, since 57s) 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: overall HEALTH_OK 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:45.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.941+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3e0c02db30 con 0x7f3e0c197d80 2024-04-24T23:36:45.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.942+0000 7f3e097fa700 1 -- 172.21.15.195:0/3672210937 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f3e04007ab0 con 0x7f3e0c197d80 2024-04-24T23:36:45.942 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-04-24T23:36:45.943 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","modified":"2024-04-24T23:36:40.330145Z","created":"2024-04-24T23:34:42.177629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 msgr2=0x7f3df803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 0x7f3df803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e00001eb0 tx=0x7f3e00005de0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 msgr2=0x7f3e0c198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 0x7f3e0c198230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bd50 tx=0x7f3e0400be30 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 shutdown_connections 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3df8039580 0x7f3df803ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e0c197d80 0x7f3e0c198230 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df40035c0 0x7f3e0c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 --2- 172.21.15.195:0/3672210937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e0c19dbb0 0x7f3e0c197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 >> 172.21.15.195:0/3672210937 conn(0x7f3e0c0fc730 msgr2=0x7f3e0c18eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 shutdown_connections 2024-04-24T23:36:45.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:45.944+0000 7f3e1226d700 1 -- 172.21.15.195:0/3672210937 wait complete. 2024-04-24T23:36:45.946 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 3 2024-04-24T23:36:46.509 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:46.510 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:46 smithi195 ceph-mon[33019]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:46.510 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/3672210937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:46.510 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:46.534 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-24T23:36:46.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-24T23:36:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:46.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:46.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:46 smithi107 ceph-mon[29118]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:46.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/3672210937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:46.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.a calling monitor election 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.c calling monitor election 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: monmap epoch 3 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: min_mon_release 19 (squid) 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: election_strategy: 1 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:47.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: fsmap 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mgrmap e13: x(active, since 57s) 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: overall HEALTH_OK 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/3672210937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T23:36:47.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:48.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mon.b calling monitor election 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mon.b calling monitor election 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mon.a calling monitor election 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mon.c calling monitor election 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: monmap epoch 3 2024-04-24T23:36:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: min_mon_release 19 (squid) 2024-04-24T23:36:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: election_strategy: 1 2024-04-24T23:36:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: fsmap 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: mgrmap e13: x(active, since 59s) 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: overall HEALTH_OK 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:48.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:48.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mon.b calling monitor election 2024-04-24T23:36:48.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mon.b calling monitor election 2024-04-24T23:36:48.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mon.a calling monitor election 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mon.c calling monitor election 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: monmap epoch 3 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: min_mon_release 19 (squid) 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: election_strategy: 1 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:48.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: fsmap 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: mgrmap e13: x(active, since 59s) 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: overall HEALTH_OK 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:48.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:48.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.b calling monitor election 2024-04-24T23:36:48.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.b calling monitor election 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.a calling monitor election 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.c calling monitor election 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: monmap epoch 3 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: last_changed 2024-04-24T23:36:40.330145+0000 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: created 2024-04-24T23:34:42.177629+0000 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: min_mon_release 19 (squid) 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: election_strategy: 1 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.c 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: fsmap 2024-04-24T23:36:48.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: mgrmap e13: x(active, since 59s) 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: overall HEALTH_OK 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:48.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:48.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/1342295907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8106be0 msgr2=0x7fb6d8106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:48.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/1342295907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8106be0 0x7fb6d8106fc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009910 tx=0x7fb6c8009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:48.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/1342295907 shutdown_connections 2024-04-24T23:36:48.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/1342295907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8106be0 0x7fb6d8106fc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/1342295907 >> 172.21.15.107:0/1342295907 conn(0x7fb6d8077800 msgr2=0x7fb6d80feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:48.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/1342295907 shutdown_connections 2024-04-24T23:36:48.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.399+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/1342295907 wait complete. 2024-04-24T23:36:48.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 Processor -- start 2024-04-24T23:36:48.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 -- start start 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d8106be0 0x7fb6d819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 0x7fb6d8197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810b1a0 con 0x7fb6d8197d50 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810b020 con 0x7fb6d8106be0 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.400+0000 7fb6dd1f3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810b320 con 0x7fb6d819dbd0 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37466/0 (socket says 172.21.15.107:37466) 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 -- 172.21.15.107:0/49377931 learned_addr learned my addr 172.21.15.107:0/49377931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6cffff700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 0x7fb6d8197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d6d9d700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d8106be0 0x7fb6d819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 -- 172.21.15.107:0/49377931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 msgr2=0x7fb6d8197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:48.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 0x7fb6d8197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 -- 172.21.15.107:0/49377931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d8106be0 msgr2=0x7fb6d819d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d8106be0 0x7fb6d819d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 -- 172.21.15.107:0/49377931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d81a44f0 con 0x7fb6d8197d50 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6cffff700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 0x7fb6d8197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d759e700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000ba50 tx=0x7fb6d000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0019070 con 0x7fb6d8197d50 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6d000ce10 con 0x7fb6d8197d50 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d81a4780 con 0x7fb6d8197d50 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0007350 con 0x7fb6d8197d50 2024-04-24T23:36:48.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.401+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d81a4c90 con 0x7fb6d8197d50 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.402+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb6d001c070 con 0x7fb6d8197d50 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.403+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d81021e0 con 0x7fb6d8197d50 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.403+0000 7fb6d4d99700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 0x7fb6c403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.403+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb6d004b710 con 0x7fb6d8197d50 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.403+0000 7fb6d6d9d700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 0x7fb6c403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:48.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.403+0000 7fb6d6d9d700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 0x7fb6c403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:48.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.408+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb6d0007dd0 con 0x7fb6d8197d50 2024-04-24T23:36:48.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.611+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb6d81a4fb0 con 0x7fb6d8197d50 2024-04-24T23:36:48.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.612+0000 7fb6d4d99700 1 -- 172.21.15.107:0/49377931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fb6d0014440 con 0x7fb6d8197d50 2024-04-24T23:36:48.612 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:48.612 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-04-24T23:36:48.613 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:36:48.613 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2024-04-24T23:36:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 msgr2=0x7fb6c403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 0x7fb6c403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c80058e0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 msgr2=0x7fb6d8198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:48.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000ba50 tx=0x7fb6d000bd60 comp rx=0 tx=0).stop 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 shutdown_connections 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6c4039580 0x7fb6c403ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6d8197d50 0x7fb6d8198200 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6d8106be0 0x7fb6d819d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 --2- 172.21.15.107:0/49377931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6d819dbd0 0x7fb6d8197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 >> 172.21.15.107:0/49377931 conn(0x7fb6d8077800 msgr2=0x7fb6d8190630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 shutdown_connections 2024-04-24T23:36:48.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:48.614+0000 7fb6dd1f3700 1 -- 172.21.15.107:0/49377931 wait complete. 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi107:/etc/ceph/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: Reconfiguring daemon mon.a on smithi107 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:49.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/49377931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi107:/etc/ceph/ceph.conf 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: Reconfiguring daemon mon.a on smithi107 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:49.314 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/49377931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi107:/etc/ceph/ceph.conf 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi170:/etc/ceph/ceph.conf 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi195:/etc/ceph/ceph.conf 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Updating smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: Reconfiguring daemon mon.a on smithi107 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:36:49.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/49377931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:49.341 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-24T23:36:49.342 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:36:49.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T23:36:49.369 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:36:49.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:49.434 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:36:49.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T23:36:49.465 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:36:49.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:49.532 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:36:49.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T23:36:49.563 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:36:49.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T23:36:49.631 INFO:tasks.cephadm:Adding mgr.x on smithi107 2024-04-24T23:36:49.632 INFO:tasks.cephadm:Adding mgr.y on smithi170 2024-04-24T23:36:49.632 INFO:tasks.cephadm:Adding mgr.z on smithi195 2024-04-24T23:36:49.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi107=x;smithi170=y;smithi195=z' 2024-04-24T23:36:49.906 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:36:51.213 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:51.214 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.214 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.214 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:51.214 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:51.214 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:51.264 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:51.264 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.264 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.264 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:51.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:51.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:51.337 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:51.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:51.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:51.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:51.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:51.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- 172.21.15.195:0/552660586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc100cd0 msgr2=0x7f3acc1010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:51.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/552660586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc100cd0 0x7f3acc1010f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009910 tx=0x7f3abc009c20 comp rx=0 tx=0).stop 2024-04-24T23:36:51.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- 172.21.15.195:0/552660586 shutdown_connections 2024-04-24T23:36:51.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/552660586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc100cd0 0x7f3acc1010f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- 172.21.15.195:0/552660586 >> 172.21.15.195:0/552660586 conn(0x7f3acc0fc270 msgr2=0x7f3acc0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:51.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- 172.21.15.195:0/552660586 shutdown_connections 2024-04-24T23:36:51.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- 172.21.15.195:0/552660586 wait complete. 2024-04-24T23:36:51.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 Processor -- start 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 -- start start 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 0x7f3acc197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.534+0000 7f3ad3a42700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 0x7f3acc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad3a42700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc106d10 con 0x7f3acc198020 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad3a42700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc106b90 con 0x7f3acc100cd0 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad3a42700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc106e90 con 0x7f3acc197ba0 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:56678/0 (socket says 172.21.15.195:56678) 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 -- 172.21.15.195:0/1225249727 learned_addr learned my addr 172.21.15.195:0/1225249727 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad1fdf700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 0x7f3acc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:51.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad17de700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 0x7f3acc197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 -- 172.21.15.195:0/1225249727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 msgr2=0x7f3acc197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 0x7f3acc197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 -- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 msgr2=0x7f3acc19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 0x7f3acc19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 -- 172.21.15.195:0/1225249727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc19fa70 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad17de700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 0x7f3acc197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad1fdf700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 0x7f3acc19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad0fdd700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b780 tx=0x7f3ac800bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8004230 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac80043d0 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc19fd60 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8011c60 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.535+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc1a02c0 con 0x7f3acc197ba0 2024-04-24T23:36:51.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.536+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f3ac801a400 con 0x7f3acc197ba0 2024-04-24T23:36:51.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.536+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3acc063890 con 0x7f3acc197ba0 2024-04-24T23:36:51.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.536+0000 7f3ac27fc700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 0x7f3ab803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:51.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.537+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3ac804bf30 con 0x7f3acc197ba0 2024-04-24T23:36:51.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.537+0000 7f3ad17de700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 0x7f3ab803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:51.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.537+0000 7f3ad17de700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 0x7f3ab803ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009750 tx=0x7f3abc0052d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:51.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.542+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3ac80036a0 con 0x7f3acc197ba0 2024-04-24T23:36:51.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.747+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi170=y;smithi195=z", "target": ["mon-mgr", ""]}) -- 0x7f3acc062670 con 0x7f3ab8039580 2024-04-24T23:36:51.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.754+0000 7f3ac27fc700 1 -- 172.21.15.195:0/1225249727 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3acc062670 con 0x7f3ab8039580 2024-04-24T23:36:51.754 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mgr update... 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 msgr2=0x7f3ab803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 0x7f3ab803ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009750 tx=0x7f3abc0052d0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 msgr2=0x7f3acc19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b780 tx=0x7f3ac800bb40 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 shutdown_connections 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3ab8039580 0x7f3ab803ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3acc198020 0x7f3acc19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc100cd0 0x7f3acc197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 --2- 172.21.15.195:0/1225249727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3acc197ba0 0x7f3acc19cc10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 >> 172.21.15.195:0/1225249727 conn(0x7f3acc0fc270 msgr2=0x7f3acc06a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 shutdown_connections 2024-04-24T23:36:51.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:36:51.756+0000 7f3ad3a42700 1 -- 172.21.15.195:0/1225249727 wait complete. 2024-04-24T23:36:52.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: Reconfiguring mon.b (monmap changed)... 2024-04-24T23:36:52.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: Reconfiguring daemon mon.b on smithi170 2024-04-24T23:36:52.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:52.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:52.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:52.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: Reconfiguring mon.b (monmap changed)... 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: Reconfiguring daemon mon.b on smithi170 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:52.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:52.347 DEBUG:teuthology.orchestra.run.smithi170:mgr.y> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.y.service 2024-04-24T23:36:52.350 DEBUG:teuthology.orchestra.run.smithi195:mgr.z> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.z.service 2024-04-24T23:36:52.352 INFO:tasks.cephadm:Deploying OSDs... 2024-04-24T23:36:52.352 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:36:52.353 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:36:52.372 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:36:52.372 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: Reconfiguring mon.b (monmap changed)... 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: Reconfiguring daemon mon.b on smithi170 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T23:36:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:36:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:52.394 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T23:36:52.394 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:52.395 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:52.395 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-24 23:35:58.327741972 +0000 2024-04-24T23:36:52.395 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-24 23:30:52.238592909 +0000 2024-04-24T23:36:52.395 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-24 23:30:52.238592909 +0000 2024-04-24T23:36:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-24 23:30:52.238592909 +0000 2024-04-24T23:36:52.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:36:52.462 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-24T23:36:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-24T23:36:52.463 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210913 s, 2.4 MB/s 2024-04-24T23:36:52.464 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:36:52.523 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:52.579 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-24 23:35:58.356741488 +0000 2024-04-24T23:36:52.580 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-24 23:30:52.585587043 +0000 2024-04-24T23:36:52.580 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-24 23:30:52.585587043 +0000 2024-04-24T23:36:52.580 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-24 23:30:52.585587043 +0000 2024-04-24T23:36:52.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:36:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-24T23:36:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-24T23:36:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000221447 s, 2.3 MB/s 2024-04-24T23:36:52.647 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:36:52.704 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-24 23:35:58.383741037 +0000 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-24 23:30:52.946580940 +0000 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-24 23:30:52.946580940 +0000 2024-04-24T23:36:52.760 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-24 23:30:52.946580940 +0000 2024-04-24T23:36:52.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:36:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-24T23:36:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-24T23:36:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000200177 s, 2.6 MB/s 2024-04-24T23:36:52.825 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:36:52.881 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-24 23:35:58.410740587 +0000 2024-04-24T23:36:52.937 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-24 23:30:53.305574870 +0000 2024-04-24T23:36:52.938 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-24 23:30:53.305574870 +0000 2024-04-24T23:36:52.938 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-24 23:30:53.305574870 +0000 2024-04-24T23:36:52.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:36:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-24T23:36:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-24T23:36:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000198747 s, 2.6 MB/s 2024-04-24T23:36:53.002 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:36:53.057 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-04-24T23:36:53.058 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:36:53.077 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:36:53.077 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-04-24T23:36:53.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:52 smithi107 ceph-mon[29118]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi170=y;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:52 smithi107 ceph-mon[29118]: Saving service mgr spec with placement smithi107=x;smithi170=y;smithi195=z;count:3 2024-04-24T23:36:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:52 smithi107 ceph-mon[29118]: Reconfiguring mon.c (monmap changed)... 2024-04-24T23:36:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:52 smithi107 ceph-mon[29118]: Reconfiguring daemon mon.c on smithi195 2024-04-24T23:36:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:52 smithi107 ceph-mon[29118]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-04-24 23:36:39.738569835 +0000 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-04-24 23:31:10.908705697 +0000 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-04-24 23:31:10.908705697 +0000 2024-04-24T23:36:53.135 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-04-24 23:31:10.908705697 +0000 2024-04-24T23:36:53.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:36:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-04-24T23:36:53.201 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-04-24T23:36:53.202 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00012915 s, 4.0 MB/s 2024-04-24T23:36:53.203 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:36:53.263 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-04-24T23:36:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:52 smithi195 ceph-mon[33019]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi170=y;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:52 smithi195 ceph-mon[33019]: Saving service mgr spec with placement smithi107=x;smithi170=y;smithi195=z;count:3 2024-04-24T23:36:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:52 smithi195 ceph-mon[33019]: Reconfiguring mon.c (monmap changed)... 2024-04-24T23:36:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:52 smithi195 ceph-mon[33019]: Reconfiguring daemon mon.c on smithi195 2024-04-24T23:36:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:52 smithi195 ceph-mon[33019]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:53.319 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:36:53.319 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-04-24 23:36:39.739569812 +0000 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-04-24 23:31:11.229698138 +0000 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-04-24 23:31:11.229698138 +0000 2024-04-24T23:36:53.320 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-04-24 23:31:11.229698138 +0000 2024-04-24T23:36:53.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:36:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:52 smithi170 ceph-mon[32714]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi170=y;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:36:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:52 smithi170 ceph-mon[32714]: Saving service mgr spec with placement smithi107=x;smithi170=y;smithi195=z;count:3 2024-04-24T23:36:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:52 smithi170 ceph-mon[32714]: Reconfiguring mon.c (monmap changed)... 2024-04-24T23:36:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:52 smithi170 ceph-mon[32714]: Reconfiguring daemon mon.c on smithi195 2024-04-24T23:36:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:52 smithi170 ceph-mon[32714]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:53.359 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-04-24T23:36:53.359 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-04-24T23:36:53.359 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000122177 s, 4.2 MB/s 2024-04-24T23:36:53.360 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:36:53.418 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:53.474 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-04-24 23:36:39.739569812 +0000 2024-04-24T23:36:53.475 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-04-24 23:31:11.555690461 +0000 2024-04-24T23:36:53.475 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-04-24 23:31:11.555690461 +0000 2024-04-24T23:36:53.475 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-04-24 23:31:11.555690461 +0000 2024-04-24T23:36:53.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:36:53.539 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-04-24T23:36:53.539 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-04-24T23:36:53.539 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000123839 s, 4.1 MB/s 2024-04-24T23:36:53.540 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:36:53.598 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-04-24T23:36:53.654 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:36:53.654 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:53.654 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-24T23:36:53.654 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:53.655 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:53.655 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-04-24 23:36:39.740569789 +0000 2024-04-24T23:36:53.655 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-04-24 23:31:11.886682667 +0000 2024-04-24T23:36:53.655 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-04-24 23:31:11.886682667 +0000 2024-04-24T23:36:53.655 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-04-24 23:31:11.886682667 +0000 2024-04-24T23:36:53.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:36:53.717 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-04-24T23:36:53.717 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-04-24T23:36:53.717 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.9441e-05 s, 5.7 MB/s 2024-04-24T23:36:53.718 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:36:53.774 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-04-24T23:36:53.774 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:36:53.793 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:36:53.793 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-04-24T23:36:53.850 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-24 23:36:30.254915718 +0000 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-24 23:31:34.085752379 +0000 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-24 23:31:34.085752379 +0000 2024-04-24T23:36:53.851 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-24 23:31:34.085752379 +0000 2024-04-24T23:36:53.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:36:53.914 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-24T23:36:53.915 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-24T23:36:53.915 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.8789e-05 s, 5.2 MB/s 2024-04-24T23:36:53.916 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:36:53.972 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-24 23:36:30.254915718 +0000 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-24 23:31:34.417745934 +0000 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-24 23:31:34.417745934 +0000 2024-04-24T23:36:54.027 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-24 23:31:34.417745934 +0000 2024-04-24T23:36:54.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:36:54.089 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-24T23:36:54.090 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-24T23:36:54.090 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.2387e-05 s, 5.5 MB/s 2024-04-24T23:36:54.091 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:36:54.146 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-24 23:36:30.254915718 +0000 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-24 23:31:34.743739604 +0000 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-24 23:31:34.743739604 +0000 2024-04-24T23:36:54.202 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-24 23:31:34.743739604 +0000 2024-04-24T23:36:54.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:36:54.264 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-24T23:36:54.264 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-24T23:36:54.264 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000114651 s, 4.5 MB/s 2024-04-24T23:36:54.265 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:36:54.323 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-04-24T23:36:54.377 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-04-24 23:36:30.255915699 +0000 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-04-24 23:31:35.081733042 +0000 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-04-24 23:31:35.081733042 +0000 2024-04-24T23:36:54.378 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-04-24 23:31:35.081733042 +0000 2024-04-24T23:36:54.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:36:54.440 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-04-24T23:36:54.440 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-04-24T23:36:54.440 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.4118e-05 s, 6.1 MB/s 2024-04-24T23:36:54.441 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:36:54.496 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/vg_nvme/lv_4... 2024-04-24T23:36:54.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T23:36:54.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:54.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: Deploying daemon mgr.z on smithi195 2024-04-24T23:36:54.522 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:54 smithi107 ceph-mon[29118]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: Deploying daemon mgr.z on smithi195 2024-04-24T23:36:54.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:54 smithi195 ceph-mon[33019]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: Deploying daemon mgr.z on smithi195 2024-04-24T23:36:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:54 smithi170 ceph-mon[32714]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:54.735 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:56.048 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:55 smithi195 systemd[1]: Starting Ceph mgr.z for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:36:56.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 podman[34858]: 2024-04-24 23:36:56.048059487 +0000 UTC m=+0.104528737 container create cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T23:36:56.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 podman[34858]: 2024-04-24 23:36:55.960995571 +0000 UTC m=+0.017464797 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:36:56.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 podman[34858]: 2024-04-24 23:36:56.242062571 +0000 UTC m=+0.298531778 container init cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T23:36:56.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 podman[34858]: 2024-04-24 23:36:56.24511728 +0000 UTC m=+0.301586487 container start cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-24T23:36:56.313 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mgr[34878]: -- 172.21.15.195:0/2660565375 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c75d1fa5a0 con 0x55c75d1cec00 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137224 s, 764 MB/s 2024-04-24T23:36:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-04-24T23:36:56.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 bash[34858]: cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 2024-04-24T23:36:56.580 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 systemd[1]: Started Ceph mgr.z for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:36:56.580 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:56.397+0000 7f3ce3007200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:36:56.580 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:56.579+0000 7f3ce3007200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:36:56.967 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:56.681+0000 7f3ce3007200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:36:57.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-04-24T23:36:57.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:57.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:57.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:57.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:57.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:57.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:57.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:57.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:56.987+0000 7f3ce3007200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:36:57.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:57.071+0000 7f3ce3007200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:36:57.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:36:57.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:36:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:36:58.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:57 smithi107 ceph-mon[29118]: Deploying daemon mgr.y on smithi170 2024-04-24T23:36:58.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:57 smithi170 ceph-mon[32714]: Deploying daemon mgr.y on smithi170 2024-04-24T23:36:58.262 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:57 smithi195 ceph-mon[33019]: Deploying daemon mgr.y on smithi170 2024-04-24T23:36:58.554 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 systemd[1]: Starting Ceph mgr.y for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:36:58.562 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:58.262+0000 7f3ce3007200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:36:58.562 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:58.430+0000 7f3ce3007200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:36:58.830 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 podman[34079]: 2024-04-24 23:36:58.457351197 +0000 UTC m=+0.018389566 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:36:58.830 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 podman[34079]: 2024-04-24 23:36:58.559027582 +0000 UTC m=+0.120065942 container create 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-24T23:36:58.830 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 podman[34079]: 2024-04-24 23:36:58.763708632 +0000 UTC m=+0.324746991 container init 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:36:58.830 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 podman[34079]: 2024-04-24 23:36:58.766097605 +0000 UTC m=+0.327135965 container start 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-24T23:36:58.830 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mgr[34098]: -- 172.21.15.170:0/483817658 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560e2b96a5a0 con 0x560e2b93f000 2024-04-24T23:36:58.831 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 bash[34079]: 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 -- 172.21.15.107:0/1402156651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c1023a0 msgr2=0x7f929c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1402156651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c1023a0 0x7f929c102800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f92900099c0 tx=0x7f9290009cd0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 -- 172.21.15.107:0/1402156651 shutdown_connections 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1402156651 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c102d40 0x7f929c1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1402156651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1402156651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c1023a0 0x7f929c102800 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 -- 172.21.15.107:0/1402156651 >> 172.21.15.107:0/1402156651 conn(0x7f929c0fc760 msgr2=0x7f929c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 -- 172.21.15.107:0/1402156651 shutdown_connections 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.870+0000 7f92a24fc700 1 -- 172.21.15.107:0/1402156651 wait complete. 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.871+0000 7f92a24fc700 1 Processor -- start 2024-04-24T23:36:58.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.871+0000 7f92a24fc700 1 -- start start 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c1023a0 0x7f929c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10cd90 con 0x7f929c102d40 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10cc10 con 0x7f929c1011a0 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10cf10 con 0x7f929c1023a0 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f929bfff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53688/0 (socket says 172.21.15.107:53688) 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f929bfff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:46546/0 (socket says 172.21.15.107:46546) 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 -- 172.21.15.107:0/1578626428 learned_addr learned my addr 172.21.15.107:0/1578626428 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f929b7fe700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c1023a0 0x7f929c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:58.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 -- 172.21.15.107:0/1578626428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c1023a0 msgr2=0x7f929c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c1023a0 0x7f929c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 -- 172.21.15.107:0/1578626428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 msgr2=0x7f929c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 -- 172.21.15.107:0/1578626428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c1a0150 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a0a99700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9294007f10 tx=0x7f929400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294004bb0 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c1a0440 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9294004d50 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940059f0 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.872+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c1a09d0 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.873+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f929401a020 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.873+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c068f20 con 0x7f929c102d40 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.873+0000 7f92997fa700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 0x7f928403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:36:58.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.873+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f929404bcc0 con 0x7f929c102d40 2024-04-24T23:36:58.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.874+0000 7f929bfff700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 0x7f928403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:36:58.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.874+0000 7f929bfff700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 0x7f928403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f929c102200 tx=0x7f928c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:36:58.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:58.878+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9294010070 con 0x7f929c102d40 2024-04-24T23:36:58.887 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:58.600+0000 7f3ce3007200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:36:58.887 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:58.887+0000 7f3ce3007200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:36:59.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mon[32714]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 systemd[1]: Started Ceph mgr.y for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:36:59.086 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:36:58.924+0000 7f7e19cf2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:36:59.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:36:59.094+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f929c0fe940 con 0x7f9284039330 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:58.980+0000 7f3ce3007200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:59.192+0000 7f3ce3007200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-mon[33019]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:58 smithi107 ceph-mon[29118]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:36:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:36:59.495 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:36:59.410+0000 7f3ce3007200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:36:59.557 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:36:59.114+0000 7f7e19cf2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:36:59.558 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:36:59.222+0000 7f7e19cf2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:36:59.558 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:36:59.557+0000 7f7e19cf2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:36:59.835 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:36:59.655+0000 7f7e19cf2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:00.062+0000 7f3ce3007200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-mon[33019]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:00.144 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:36:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:00.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:00.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-mon[32714]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:00.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:00.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:00.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:36:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:00.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:00.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:59 smithi107 ceph-mon[29118]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:00.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:00.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:00.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:36:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:00.794 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:00.500+0000 7f3ce3007200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:37:00.794 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:00.599+0000 7f3ce3007200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:37:00.794 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:00.794+0000 7f3ce3007200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:37:01.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:01.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:37:01.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:00.902+0000 7f7e19cf2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:01.069+0000 7f7e19cf2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:01.238+0000 7f7e19cf2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:01.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:37:01.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:37:01.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:01.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T23:37:01.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T23:37:01.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:01.562 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.310+0000 7f3ce3007200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:37:01.562 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.399+0000 7f3ce3007200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:37:01.817 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:01.522+0000 7f7e19cf2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:37:01.817 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:01.614+0000 7f7e19cf2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:37:01.817 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:01.816+0000 7f7e19cf2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:37:01.970 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.635+0000 7f3ce3007200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:37:01.970 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.765+0000 7f3ce3007200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:37:01.970 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.858+0000 7f3ce3007200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:37:01.970 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:01.970+0000 7f3ce3007200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:37:02.085 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:02.006+0000 7f7e19cf2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: Reconfiguring daemon mgr.x on smithi107 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]: dispatch 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]': finished 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T23:37:02.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:01 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:02.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:02.060+0000 7f3ce3007200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:37:02.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T23:37:02.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: Reconfiguring daemon mgr.x on smithi107 2024-04-24T23:37:02.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]: dispatch 2024-04-24T23:37:02.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]': finished 2024-04-24T23:37:02.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T23:37:02.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:01 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: Reconfiguring daemon mgr.x on smithi107 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]: dispatch 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3219142649' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adc01bb-2472-44e3-abc7-5ef350338376"}]': finished 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T23:37:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:01 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:02.980 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:02.598+0000 7f7e19cf2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:02 smithi170 ceph-mon[32714]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:02 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/164085742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:03.017+0000 7f7e19cf2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:03.111+0000 7f7e19cf2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:37:03.310 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:03.309+0000 7f7e19cf2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:37:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:02 smithi195 ceph-mon[33019]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:02 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/164085742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:03.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:02.857+0000 7f3ce3007200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:37:03.312 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z[34874]: 2024-04-24T23:37:03.067+0000 7f3ce3007200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:37:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:02 smithi107 ceph-mon[29118]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:02 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/164085742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:03.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:03.987+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f92940133f0 con 0x7f929c102d40 2024-04-24T23:37:04.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-mon[33019]: Standby manager daemon z started 2024-04-24T23:37:04.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T23:37:04.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:04.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T23:37:04.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:03 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:04.085 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:03.829+0000 7f7e19cf2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:03.919+0000 7f7e19cf2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-mon[32714]: Standby manager daemon z started 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T23:37:04.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:03 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:04.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:03 smithi107 ceph-mon[29118]: Standby manager daemon z started 2024-04-24T23:37:04.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:03 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T23:37:04.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:03 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:04.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:03 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T23:37:04.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:03 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.195:0/1807689844' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:04.487 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:04.155+0000 7f7e19cf2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:37:04.488 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:04.283+0000 7f7e19cf2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:37:04.488 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:04.375+0000 7f7e19cf2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:37:04.488 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:04.486+0000 7f7e19cf2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:37:04.835 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:04.575+0000 7f7e19cf2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:37:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:04 smithi195 ceph-mon[33019]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:04 smithi195 ceph-mon[33019]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-24T23:37:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:04 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T23:37:05.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-mon[32714]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:05.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-mon[32714]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-24T23:37:05.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:04 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T23:37:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:04 smithi107 ceph-mon[29118]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:04 smithi107 ceph-mon[29118]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-24T23:37:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:04 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T23:37:05.836 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:05.374+0000 7f7e19cf2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:37:05.836 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y[34094]: 2024-04-24T23:37:05.582+0000 7f7e19cf2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:37:06.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:05.998+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f9294004f00 con 0x7f929c102d40 2024-04-24T23:37:06.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:05 smithi107 ceph-mon[29118]: Standby manager daemon y started 2024-04-24T23:37:06.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:05 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T23:37:06.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:05 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:06.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:05 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T23:37:06.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:05 smithi107 ceph-mon[29118]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:05 smithi195 ceph-mon[33019]: Standby manager daemon y started 2024-04-24T23:37:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:05 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T23:37:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:05 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:05 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T23:37:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:05 smithi195 ceph-mon[33019]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-mon[32714]: Standby manager daemon y started 2024-04-24T23:37:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T23:37:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T23:37:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T23:37:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:05 smithi170 ceph-mon[32714]: from='mgr.? 172.21.15.170:0/1194620452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T23:37:07.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:06 smithi170 ceph-mon[32714]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:07.359 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:06 smithi170 ceph-mon[32714]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-04-24T23:37:07.359 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:06 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T23:37:07.360 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:06 smithi195 ceph-mon[33019]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:07.360 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:06 smithi195 ceph-mon[33019]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-04-24T23:37:07.361 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:06 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T23:37:07.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:06 smithi107 ceph-mon[29118]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:07.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:06 smithi107 ceph-mon[29118]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-04-24T23:37:07.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:06 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T23:37:08.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:08.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:08.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:08.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:08.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:09.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:09 smithi107 ceph-mon[29118]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:09 smithi195 ceph-mon[33019]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:09 smithi170 ceph-mon[32714]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:10.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T23:37:10.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:10.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T23:37:10.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:10.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T23:37:10.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:11 smithi107 ceph-mon[29118]: Deploying daemon osd.0 on smithi107 2024-04-24T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:11 smithi107 ceph-mon[29118]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:11 smithi195 ceph-mon[33019]: Deploying daemon osd.0 on smithi107 2024-04-24T23:37:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:11 smithi195 ceph-mon[33019]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:11 smithi170 ceph-mon[32714]: Deploying daemon osd.0 on smithi107 2024-04-24T23:37:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:11 smithi170 ceph-mon[32714]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:13.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:13 smithi107 ceph-mon[29118]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:13 smithi195 ceph-mon[33019]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:13 smithi170 ceph-mon[32714]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:15.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:15 smithi107 ceph-mon[29118]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:15.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:15 smithi195 ceph-mon[33019]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:15 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:15 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:15 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:15 smithi170 ceph-mon[32714]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:15 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:15 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:15.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:15 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:17.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:17 smithi107 ceph-mon[29118]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:17 smithi195 ceph-mon[33019]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:17.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:17 smithi170 ceph-mon[32714]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.451+0000 7f92997fa700 1 -- 172.21.15.107:0/1578626428 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f929c0fe940 con 0x7f9284039330 2024-04-24T23:37:18.452 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 msgr2=0x7f928403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 0x7f928403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f929c102200 tx=0x7f928c00b410 comp rx=0 tx=0).stop 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 msgr2=0x7f929c19fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9294007f10 tx=0x7f929400d3b0 comp rx=0 tx=0).stop 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 shutdown_connections 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9284039330 0x7f928403b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c102d40 0x7f929c19fab0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c1011a0 0x7f929c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 --2- 172.21.15.107:0/1578626428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f929c1023a0 0x7f929c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 >> 172.21.15.107:0/1578626428 conn(0x7f929c0fc760 msgr2=0x7f929c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:37:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 shutdown_connections 2024-04-24T23:37:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:37:18.454+0000 7f92a24fc700 1 -- 172.21.15.107:0/1578626428 wait complete. 2024-04-24T23:37:18.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:18.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:19.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:19.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:19.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:19.092 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.0.service 2024-04-24T23:37:19.094 INFO:tasks.cephadm:Deploying osd.1 on smithi170 with /dev/vg_nvme/lv_4... 2024-04-24T23:37:19.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T23:37:19.335 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:37:19.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:19 smithi170 ceph-mon[32714]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T23:37:19.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:19 smithi107 ceph-mon[29118]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T23:37:20.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:19 smithi195 ceph-mon[33019]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T23:37:20.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:20 smithi170 ceph-mon[32714]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T23:37:20.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:20 smithi170 ceph-mon[32714]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T23:37:20.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:20 smithi170 ceph-mon[32714]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:37:20.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:20.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:20 smithi170 ceph-mon[32714]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:20.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-mon[29118]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T23:37:20.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-mon[29118]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T23:37:20.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-mon[29118]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:37:20.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:20.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-mon[29118]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:20.879 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0[41364]: 2024-04-24T23:37:20.648+0000 7f3b9c876700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T23:37:20.879 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:37:20 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0[41364]: 2024-04-24T23:37:20.651+0000 7f3b98669700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836739 s, 1.3 GB/s 2024-04-24T23:37:20.984 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-04-24T23:37:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:20 smithi195 ceph-mon[33019]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T23:37:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:20 smithi195 ceph-mon[33019]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T23:37:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:20 smithi195 ceph-mon[33019]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:37:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:20 smithi195 ceph-mon[33019]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:21.661 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-04-24T23:37:21.899 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:37:22.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:21 smithi107 ceph-mon[29118]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:37:22.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:21 smithi107 ceph-mon[29118]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T23:37:22.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:21 smithi170 ceph-mon[32714]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:37:22.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:21 smithi170 ceph-mon[32714]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T23:37:22.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:21 smithi195 ceph-mon[33019]: from='osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:37:22.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:21 smithi195 ceph-mon[33019]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T23:37:22.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.912 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:37:22.912 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] boot 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi107 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi107 to 17088M 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:22.913 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:22.914 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:22.914 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:37:23.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:37:23.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] boot 2024-04-24T23:37:23.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T23:37:23.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi107 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi107 to 17088M 2024-04-24T23:37:23.064 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.065 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:23.065 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:23.065 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:37:23.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:37:23.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: osd.0 [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] boot 2024-04-24T23:37:23.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi107 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:23.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi107 to 17088M 2024-04-24T23:37:23.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:23.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:23.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:23.412 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- 172.21.15.170:0/4013229387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45901027f0 msgr2=0x7f4590102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 --2- 172.21.15.170:0/4013229387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45901027f0 0x7f4590102c10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4584009990 tx=0x7f4584009ca0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- 172.21.15.170:0/4013229387 shutdown_connections 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 --2- 172.21.15.170:0/4013229387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101e10 0x7f45901096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 --2- 172.21.15.170:0/4013229387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45901027f0 0x7f4590102c10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 --2- 172.21.15.170:0/4013229387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4590101450 0x7f45901018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- 172.21.15.170:0/4013229387 >> 172.21.15.170:0/4013229387 conn(0x7f45900fbf00 msgr2=0x7f45900fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- 172.21.15.170:0/4013229387 shutdown_connections 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- 172.21.15.170:0/4013229387 wait complete. 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 Processor -- start 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.410+0000 7f4596baf700 1 -- start start 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 0x7f4590100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45901027f0 0x7f459019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f459010ce90 con 0x7f4590101450 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f459010cd10 con 0x7f4590101e10 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f459010d010 con 0x7f45901027f0 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:45502/0 (socket says 172.21.15.170:45502) 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 -- 172.21.15.170:0/4261229049 learned_addr learned my addr 172.21.15.170:0/4261229049 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f459514c700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45901027f0 0x7f459019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f459494b700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 0x7f4590100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 -- 172.21.15.170:0/4261229049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45901027f0 msgr2=0x7f459019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45901027f0 0x7f459019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 -- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 msgr2=0x7f4590100ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:23.414 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 0x7f4590100ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 -- 172.21.15.170:0/4261229049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45901a0240 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f459494b700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 0x7f4590100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458ffff700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b530 tx=0x7f457c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c015020 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457c011d10 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45901a0530 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.411+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c010e50 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.412+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45901a0af0 con 0x7f4590101e10 2024-04-24T23:37:23.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.412+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4590068f20 con 0x7f4590101e10 2024-04-24T23:37:23.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.413+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f457c018020 con 0x7f4590101e10 2024-04-24T23:37:23.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.414+0000 7f458dffb700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 0x7f45800a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:23.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.414+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f457c0ccdb0 con 0x7f4590101e10 2024-04-24T23:37:23.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.414+0000 7f459494b700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 0x7f45800a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:23.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.415+0000 7f459494b700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 0x7f45800a5270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45840039c0 tx=0x7f4584005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:37:23.419 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.419+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f457c01f020 con 0x7f4590101e10 2024-04-24T23:37:23.623 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:23.623+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4590062670 con 0x7f45800a2db0 2024-04-24T23:37:24.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:23 smithi195 ceph-mon[33019]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T23:37:24.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:24.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:24.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:24.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:23 smithi170 ceph-mon[32714]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T23:37:24.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:24.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:24.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:24.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:23 smithi107 ceph-mon[29118]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T23:37:24.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:24.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:24.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:24.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:24 smithi170 ceph-mon[32714]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:24.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:24 smithi170 ceph-mon[32714]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:25.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:24 smithi195 ceph-mon[33019]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:25.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:24 smithi195 ceph-mon[33019]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:25.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:24 smithi107 ceph-mon[29118]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:25.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:24 smithi107 ceph-mon[29118]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/1315935642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]': finished 2024-04-24T23:37:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T23:37:27.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:27.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:26 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/606761847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/1315935642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]': finished 2024-04-24T23:37:27.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T23:37:27.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:27.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:26 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/606761847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/1315935642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]: dispatch 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9087e7-4eaa-4516-b08f-3eef8dbae2d5"}]': finished 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:26 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/606761847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:28 smithi195 ceph-mon[33019]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:28 smithi170 ceph-mon[32714]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:29.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:28 smithi107 ceph-mon[29118]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:31.223 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:30 smithi170 ceph-mon[32714]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:31.304 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:30 smithi195 ceph-mon[33019]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:30 smithi107 ceph-mon[29118]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:32 smithi195 ceph-mon[33019]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:32 smithi170 ceph-mon[32714]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:33.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:32 smithi107 ceph-mon[29118]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:34.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T23:37:34.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:34.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:33 smithi170 ceph-mon[32714]: Deploying daemon osd.1 on smithi170 2024-04-24T23:37:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T23:37:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:33 smithi195 ceph-mon[33019]: Deploying daemon osd.1 on smithi170 2024-04-24T23:37:34.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T23:37:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:33 smithi107 ceph-mon[29118]: Deploying daemon osd.1 on smithi170 2024-04-24T23:37:35.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:34 smithi170 ceph-mon[32714]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:35.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:34 smithi195 ceph-mon[33019]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:35.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:34 smithi107 ceph-mon[29118]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:37.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:36 smithi170 ceph-mon[32714]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:37.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:36 smithi195 ceph-mon[33019]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:36 smithi107 ceph-mon[29118]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:39.027 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:38 smithi170 ceph-mon[32714]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:39.027 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:39.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:39.028 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:39.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:38 smithi195 ceph-mon[33019]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:39.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:39.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:39.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:38 smithi107 ceph-mon[29118]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:41.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:40 smithi195 ceph-mon[33019]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:41.063 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:40 smithi170 ceph-mon[32714]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:40 smithi107 ceph-mon[29118]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:41.985 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.984+0000 7f458dffb700 1 -- 172.21.15.170:0/4261229049 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4590062670 con 0x7f45800a2db0 2024-04-24T23:37:41.985 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 1 on host 'smithi170' 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 msgr2=0x7f45800a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 0x7f45800a5270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45840039c0 tx=0x7f4584005fb0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 msgr2=0x7f459019d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b530 tx=0x7f457c00b8f0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 shutdown_connections 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f45800a2db0 0x7f45800a5270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f45901027f0 0x7f459019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4590101450 0x7f4590100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.987+0000 7f4596baf700 1 --2- 172.21.15.170:0/4261229049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4590101e10 0x7f459019d330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:41.991 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.988+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 >> 172.21.15.170:0/4261229049 conn(0x7f45900fbf00 msgr2=0x7f45901057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:37:41.992 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.988+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 shutdown_connections 2024-04-24T23:37:41.992 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:37:41.988+0000 7f4596baf700 1 -- 172.21.15.170:0/4261229049 wait complete. 2024-04-24T23:37:42.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:42.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:42.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:42.588 DEBUG:teuthology.orchestra.run.smithi170:osd.1> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.1.service 2024-04-24T23:37:42.590 INFO:tasks.cephadm:Deploying osd.2 on smithi195 with /dev/vg_nvme/lv_4... 2024-04-24T23:37:42.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T23:37:42.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:42.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:42.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:42.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:37:42.832 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:37:43.261 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:43 smithi195 ceph-mon[33019]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:43.262 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:43 smithi195 ceph-mon[33019]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:43.262 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:43 smithi195 ceph-mon[33019]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:43.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:43 smithi170 ceph-mon[32714]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:43.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:43 smithi170 ceph-mon[32714]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:43.435 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:43 smithi170 ceph-mon[32714]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:43.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:43 smithi107 ceph-mon[29118]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:43.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:43 smithi107 ceph-mon[29118]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:43.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:43 smithi107 ceph-mon[29118]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T23:37:44.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T23:37:44.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T23:37:44.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:44.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:44 smithi195 ceph-mon[33019]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-mon[32714]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:37:44.336 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1[37311]: 2024-04-24T23:37:44.160+0000 7f8750076700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T23:37:44.336 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:37:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1[37311]: 2024-04-24T23:37:44.163+0000 7f874be69700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839673 s, 1.2 GB/s 2024-04-24T23:37:44.615 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: from='osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:37:44.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:44 smithi107 ceph-mon[29118]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:37:45.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_4 2024-04-24T23:37:45.230 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:45 smithi195 ceph-mon[33019]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:45.231 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:45 smithi195 ceph-mon[33019]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T23:37:45.231 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.231 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.231 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:45 smithi170 ceph-mon[32714]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:45 smithi170 ceph-mon[32714]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T23:37:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.444 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:37:45.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:45 smithi107 ceph-mon[29118]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:45.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:45 smithi107 ceph-mon[29118]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T23:37:45.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:45.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:46.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:37:46.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:37:46.435 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] boot 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:46.436 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] boot 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:46.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:46.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: osd.1 [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] boot 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:37:46.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:37:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 -- 172.21.15.195:0/1677300312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101ee0 msgr2=0x7f69481097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:46.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 --2- 172.21.15.195:0/1677300312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101ee0 0x7f69481097a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f694000b0d0 tx=0x7f694000b3e0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 -- 172.21.15.195:0/1677300312 shutdown_connections 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 --2- 172.21.15.195:0/1677300312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101ee0 0x7f69481097a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 --2- 172.21.15.195:0/1677300312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f6948102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 --2- 172.21.15.195:0/1677300312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948101540 0x7f69481019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 -- 172.21.15.195:0/1677300312 >> 172.21.15.195:0/1677300312 conn(0x7f69480fbf90 msgr2=0x7f69480fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 -- 172.21.15.195:0/1677300312 shutdown_connections 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.931+0000 7f694f6d6700 1 -- 172.21.15.195:0/1677300312 wait complete. 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 Processor -- start 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 -- start start 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948101ee0 0x7f69481989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f694819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f694810cf20 con 0x7f6948101ee0 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f694810cda0 con 0x7f6948102800 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694f6d6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f694810d0a0 con 0x7f6948101540 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:44898/0 (socket says 172.21.15.195:44898) 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 -- 172.21.15.195:0/2336709704 learned_addr learned my addr 172.21.15.195:0/2336709704 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694dc73700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f694819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 -- 172.21.15.195:0/2336709704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 msgr2=0x7f694819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f694819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 -- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948101ee0 msgr2=0x7f69481989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948101ee0 0x7f69481989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694d472700 1 -- 172.21.15.195:0/2336709704 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69481a0110 con 0x7f6948101540 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.932+0000 7f694dc73700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f694819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:37:46.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f694d472700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f694400b530 tx=0x7f694400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6944015020 con 0x7f6948101540 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69481a0400 con 0x7f6948101540 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6944011d10 con 0x7f6948101540 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694400f2e0 con 0x7f6948101540 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.933+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69481a0940 con 0x7f6948101540 2024-04-24T23:37:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.934+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6948068f20 con 0x7f6948101540 2024-04-24T23:37:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.935+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f6944018020 con 0x7f6948101540 2024-04-24T23:37:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.935+0000 7f693e7fc700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 0x7f69340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:37:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.935+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f69440cd1d0 con 0x7f6948101540 2024-04-24T23:37:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.936+0000 7f694cc71700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 0x7f69340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:37:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.936+0000 7f694cc71700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 0x7f69340a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6938005fd0 tx=0x7f6938005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:37:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:46.939+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f694401f020 con 0x7f6948101540 2024-04-24T23:37:47.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:37:47.146+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6948062670 con 0x7f69340a2b90 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi170 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi170 to 17088M 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi170 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi170 to 17088M 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:47.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi170 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi170 to 17088M 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:37:47.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:48.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:48 smithi195 ceph-mon[33019]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:48 smithi170 ceph-mon[32714]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:48 smithi107 ceph-mon[29118]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:37:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:49 smithi195 ceph-mon[33019]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:49.466 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:49 smithi170 ceph-mon[32714]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:49 smithi107 ceph-mon[29118]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:50 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/3347992545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:50 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:50 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]': finished 2024-04-24T23:37:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:50 smithi195 ceph-mon[33019]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T23:37:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:37:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:50 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/3347992545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:50 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:50 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]': finished 2024-04-24T23:37:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:50 smithi107 ceph-mon[29118]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T23:37:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:37:50.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:50 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/3347992545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:50 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]: dispatch 2024-04-24T23:37:50.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:50 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "010d750a-e582-46e0-ac12-95fedac8d734"}]': finished 2024-04-24T23:37:50.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:50 smithi170 ceph-mon[32714]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T23:37:50.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:37:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:51 smithi195 ceph-mon[33019]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:51 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/4016315069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:51 smithi170 ceph-mon[32714]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:51 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/4016315069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:51.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:51 smithi107 ceph-mon[29118]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:51.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:51 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/4016315069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:37:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:53 smithi195 ceph-mon[33019]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:53 smithi170 ceph-mon[32714]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:53.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:53 smithi107 ceph-mon[29118]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:55.459 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:55 smithi195 ceph-mon[33019]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:55 smithi170 ceph-mon[32714]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:55.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:55 smithi107 ceph-mon[29118]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:57.476 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:57 smithi195 ceph-mon[33019]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:57.476 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:57 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T23:37:57.476 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:57 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:57 smithi170 ceph-mon[32714]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:57 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T23:37:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:57 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:57 smithi107 ceph-mon[29118]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:57 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T23:37:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:57 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:37:58.473 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:58 smithi195 ceph-mon[33019]: Deploying daemon osd.2 on smithi195 2024-04-24T23:37:58.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:58 smithi170 ceph-mon[32714]: Deploying daemon osd.2 on smithi195 2024-04-24T23:37:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:58 smithi107 ceph-mon[29118]: Deploying daemon osd.2 on smithi195 2024-04-24T23:37:59.526 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:37:59 smithi195 ceph-mon[33019]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:37:59 smithi170 ceph-mon[32714]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:37:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:37:59 smithi107 ceph-mon[29118]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:00.526 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:00 smithi195 ceph-mon[33019]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:00 smithi170 ceph-mon[32714]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:00 smithi107 ceph-mon[29118]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:03.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:03.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:02 smithi195 ceph-mon[33019]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:03.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:03.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:02 smithi170 ceph-mon[32714]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:03.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:03.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:03.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:02 smithi107 ceph-mon[29118]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:05.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.207+0000 7f693e7fc700 1 -- 172.21.15.195:0/2336709704 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6948062670 con 0x7f69340a2b90 2024-04-24T23:38:05.208 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 2 on host 'smithi195' 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:05.209 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.209+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 msgr2=0x7f69340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 0x7f69340a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6938005fd0 tx=0x7f6938005d00 comp rx=0 tx=0).stop 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 msgr2=0x7f69481984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f694400b530 tx=0x7f694400b8f0 comp rx=0 tx=0).stop 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 shutdown_connections 2024-04-24T23:38:05.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69340a2b90 0x7f69340a5050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6948102800 0x7f694819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6948101540 0x7f69481984b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 --2- 172.21.15.195:0/2336709704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948101ee0 0x7f69481989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 >> 172.21.15.195:0/2336709704 conn(0x7f69480fbf90 msgr2=0x7f69480fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 shutdown_connections 2024-04-24T23:38:05.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:38:05.210+0000 7f694f6d6700 1 -- 172.21.15.195:0/2336709704 wait complete. 2024-04-24T23:38:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:05.833 DEBUG:teuthology.orchestra.run.smithi195:osd.2> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.2.service 2024-04-24T23:38:05.875 INFO:tasks.cephadm:Deploying osd.3 on smithi107 with /dev/vg_nvme/lv_3... 2024-04-24T23:38:05.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T23:38:06.120 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:38:06.352 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:06 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.352 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:06 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:06.352 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:06 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.352 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:06 smithi195 ceph-mon[33019]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:06.352 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:06 smithi195 ceph-mon[33019]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:06.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:06 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:06 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:06.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:06 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:06 smithi107 ceph-mon[29118]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:06.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:06 smithi107 ceph-mon[29118]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:06.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:06 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:06 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:06.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:06 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:06.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:06 smithi170 ceph-mon[32714]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:06.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:06 smithi170 ceph-mon[32714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T23:38:07.305 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:07.306 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T23:38:07.306 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T23:38:07.306 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:07.306 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:07.306 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-mon[33019]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:07.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:07.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T23:38:07.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T23:38:07.466 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:07.466 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:07.466 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:07 smithi107 ceph-mon[29118]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T23:38:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T23:38:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: from='osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:07.488 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:07 smithi170 ceph-mon[32714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:38:08.045 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T23:38:08.046 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T23:38:08.046 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-04-24T23:38:08.046 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-04-24T23:38:08.046 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114029 s, 920 MB/s 2024-04-24T23:38:08.046 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-04-24T23:38:08.062 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2[38138]: 2024-04-24T23:38:07.678+0000 7f3cf6f23700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T23:38:08.062 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2[38138]: 2024-04-24T23:38:07.682+0000 7f3cee50d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:38:08.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:08 smithi195 ceph-mon[33019]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:38:08.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:08 smithi195 ceph-mon[33019]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T23:38:08.437 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.438 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.438 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:08 smithi195 ceph-mon[33019]: from='osd.2 ' entity='osd.2' 2024-04-24T23:38:08.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:08 smithi170 ceph-mon[32714]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:38:08.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:08 smithi170 ceph-mon[32714]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T23:38:08.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:08 smithi170 ceph-mon[32714]: from='osd.2 ' entity='osd.2' 2024-04-24T23:38:08.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:08 smithi107 ceph-mon[29118]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:38:08.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:08 smithi107 ceph-mon[29118]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T23:38:08.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:08.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:08 smithi107 ceph-mon[29118]: from='osd.2 ' entity='osd.2' 2024-04-24T23:38:08.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-04-24T23:38:08.968 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] boot 2024-04-24T23:38:09.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:09 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] boot 2024-04-24T23:38:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:09.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:09 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:38:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:38:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-24T23:38:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: osd.2 [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] boot 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:09 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:10.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:10 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi195 2024-04-24T23:38:10.423 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:10 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi195 to 17088M 2024-04-24T23:38:10.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:10 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi195 2024-04-24T23:38:10.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:10 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi195 to 17088M 2024-04-24T23:38:10.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:10 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi195 2024-04-24T23:38:10.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:10 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi195 to 17088M 2024-04-24T23:38:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 -- 172.21.15.107:0/2268207194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4102d20 msgr2=0x7fcec41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 --2- 172.21.15.107:0/2268207194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4102d20 0x7fcec41095d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00b0d0 tx=0x7fcebc00b3e0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 -- 172.21.15.107:0/2268207194 shutdown_connections 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 --2- 172.21.15.107:0/2268207194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4102d20 0x7fcec41095d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 --2- 172.21.15.107:0/2268207194 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4101180 0x7fcec41015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 --2- 172.21.15.107:0/2268207194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 0x7fcec41027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 -- 172.21.15.107:0/2268207194 >> 172.21.15.107:0/2268207194 conn(0x7fcec40fc720 msgr2=0x7fcec40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 -- 172.21.15.107:0/2268207194 shutdown_connections 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.635+0000 7fcec9367700 1 -- 172.21.15.107:0/2268207194 wait complete. 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 Processor -- start 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 -- start start 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 0x7fcec41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4102d20 0x7fcec419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410cde0 con 0x7fcec4101180 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410cc60 con 0x7fcec4102380 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec9367700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410cf60 con 0x7fcec4102d20 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec2ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec2ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52620/0 (socket says 172.21.15.107:52620) 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec2ffd700 1 -- 172.21.15.107:0/2300833345 learned_addr learned my addr 172.21.15.107:0/2300833345 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec27fc700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 0x7fcec41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.636+0000 7fcec37fe700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4102d20 0x7fcec419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 -- 172.21.15.107:0/2300833345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4102d20 msgr2=0x7fcec419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4102d20 0x7fcec419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 -- 172.21.15.107:0/2300833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 msgr2=0x7fcec41989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 0x7fcec41989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 -- 172.21.15.107:0/2300833345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec41a0110 con 0x7fcec4101180 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec2ffd700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b530 tx=0x7fceb400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:38:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4015020 con 0x7fcec4101180 2024-04-24T23:38:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec41a0400 con 0x7fcec4101180 2024-04-24T23:38:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb4011d10 con 0x7fcec4101180 2024-04-24T23:38:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4010d90 con 0x7fcec4101180 2024-04-24T23:38:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.637+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a09c0 con 0x7fcec4101180 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.638+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec4068f20 con 0x7fcec4101180 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.639+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fceb4018020 con 0x7fcec4101180 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.640+0000 7fcebbfff700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 0x7fceb00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.640+0000 7fcec27fc700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 0x7fceb00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.640+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fceb40cd7f0 con 0x7fcec4101180 2024-04-24T23:38:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.640+0000 7fcec27fc700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 0x7fceb00a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fceac005950 tx=0x7fceac00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:38:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.644+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fceb401f020 con 0x7fcec4101180 2024-04-24T23:38:10.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:10.846+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fcec41a0c80 con 0x7fceb00a2b90 2024-04-24T23:38:11.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: pgmap v79: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:11.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:11.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T23:38:11.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:11.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:11.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:11.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: pgmap v79: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:11.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:11.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:11.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: pgmap v79: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:11.825 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46043]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:11.825 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46043]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:11.825 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46043]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:11.825 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46043]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:12.128 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:12.128 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:12.128 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46063]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:12.128 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:38:11 smithi107 sudo[46063]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:12.346 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:12.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:12 smithi170 ceph-mon[32714]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T23:38:12.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:12.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:12 smithi170 ceph-mon[32714]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:12.347 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41001]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:12.347 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:12.347 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41001]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:12.347 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41001]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:12.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:12.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:12 smithi107 ceph-mon[29118]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T23:38:12.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:12.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:12 smithi107 ceph-mon[29118]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:12.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:38:12.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:12 smithi195 ceph-mon[33019]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T23:38:12.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:12.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:12 smithi195 ceph-mon[33019]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:12.835 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41004]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:12.836 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:12.836 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41004]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:12.836 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:38:12 smithi170 sudo[41004]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:12.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:12.863+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fceb4097b80 con 0x7fcec4101180 2024-04-24T23:38:12.865 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:12.865 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:12.865 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:12.865 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41764]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:13.312 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:13.312 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:13.312 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41768]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:13.312 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:38:12 smithi195 sudo[41768]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:13.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46100]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:13.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:13.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46100]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:13.404 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46100]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 ceph-mon[33019]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 ceph-mon[33019]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41771]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41771]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41771]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:13.663 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41771]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:13.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:13 smithi170 ceph-mon[32714]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T23:38:13.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:13 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:13 smithi170 ceph-mon[32714]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T23:38:13.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:13 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:13 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46123]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46123]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46123]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 sudo[46123]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 ceph-mon[29118]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 ceph-mon[29118]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T23:38:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:13.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:13 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41774]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:14.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41774]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:14.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41774]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:14.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:13 smithi195 sudo[41774]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:14.395 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T23:38:14.395 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:14.395 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41007]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:14.396 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41007]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:14.396 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41010]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]': finished 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:14.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:14 smithi107 ceph-mon[29118]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]': finished 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:14.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:14 smithi195 ceph-mon[33019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41010]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41010]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 sudo[41010]: pam_unix(sudo:session): session closed for user root 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]: dispatch 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/675243536' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14800c24-129e-4081-bb09-cbef9af89c78"}]': finished 2024-04-24T23:38:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:14.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:14 smithi170 ceph-mon[32714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T23:38:15.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:15 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2582752727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:15.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:15 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2582752727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:15.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:15 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2582752727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:16 smithi195 ceph-mon[33019]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:16.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:16 smithi170 ceph-mon[32714]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:16.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:16 smithi107 ceph-mon[29118]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:19.118 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:18 smithi107 ceph-mon[29118]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:18 smithi195 ceph-mon[33019]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:18 smithi170 ceph-mon[32714]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:21.168 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:20 smithi107 ceph-mon[29118]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:21.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:20 smithi195 ceph-mon[33019]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:21.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:20 smithi170 ceph-mon[32714]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:22.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T23:38:22.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:22.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:21 smithi107 ceph-mon[29118]: Deploying daemon osd.3 on smithi107 2024-04-24T23:38:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T23:38:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:22.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:21 smithi195 ceph-mon[33019]: Deploying daemon osd.3 on smithi107 2024-04-24T23:38:22.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T23:38:22.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:22.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:21 smithi170 ceph-mon[32714]: Deploying daemon osd.3 on smithi107 2024-04-24T23:38:23.093 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:22 smithi107 ceph-mon[29118]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:22 smithi195 ceph-mon[33019]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:23.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:22 smithi170 ceph-mon[32714]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:24 smithi195 ceph-mon[33019]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:25.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:24 smithi170 ceph-mon[32714]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:25.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:24 smithi107 ceph-mon[29118]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:27.240 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:26 smithi107 ceph-mon[29118]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:27.240 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:26 smithi195 ceph-mon[33019]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:26 smithi170 ceph-mon[32714]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:29.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:28 smithi107 ceph-mon[29118]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:28 smithi195 ceph-mon[33019]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:28 smithi170 ceph-mon[32714]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:30.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.310+0000 7fcebbfff700 1 -- 172.21.15.107:0/2300833345 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcec41a0c80 con 0x7fceb00a2b90 2024-04-24T23:38:30.310 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 3 on host 'smithi107' 2024-04-24T23:38:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 msgr2=0x7fceb00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 0x7fceb00a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fceac005950 tx=0x7fceac00a300 comp rx=0 tx=0).stop 2024-04-24T23:38:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 msgr2=0x7fcec41984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:30.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b530 tx=0x7fceb400b840 comp rx=0 tx=0).stop 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 shutdown_connections 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fceb00a2b90 0x7fceb00a5050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcec4102d20 0x7fcec419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcec4101180 0x7fcec41984b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 --2- 172.21.15.107:0/2300833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcec4102380 0x7fcec41989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 >> 172.21.15.107:0/2300833345 conn(0x7fcec40fc720 msgr2=0x7fcec40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 shutdown_connections 2024-04-24T23:38:30.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:38:30.312+0000 7fcec9367700 1 -- 172.21.15.107:0/2300833345 wait complete. 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:30.695 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.695 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.695 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:30.695 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:30.696 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.696 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:30.696 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.696 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.696 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:30.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:31.020 DEBUG:teuthology.orchestra.run.smithi107:osd.3> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.3.service 2024-04-24T23:38:31.022 INFO:tasks.cephadm:Deploying osd.4 on smithi170 with /dev/vg_nvme/lv_3... 2024-04-24T23:38:31.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T23:38:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:38:31.573 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:31 smithi107 ceph-mon[29118]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T23:38:31.629 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:31 smithi170 ceph-mon[32714]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T23:38:31.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:31 smithi195 ceph-mon[33019]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T23:38:32.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-mon[29118]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T23:38:32.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-mon[29118]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T23:38:32.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-mon[29118]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:38:32.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:32.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-mon[29118]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:32.657 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:32 smithi170 ceph-mon[32714]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T23:38:32.657 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:32 smithi170 ceph-mon[32714]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T23:38:32.657 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:32 smithi170 ceph-mon[32714]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:38:32.657 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:32 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:32.657 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:32 smithi170 ceph-mon[32714]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:32.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:32 smithi195 ceph-mon[33019]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T23:38:32.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:32 smithi195 ceph-mon[33019]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T23:38:32.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:32 smithi195 ceph-mon[33019]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:38:32.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:32 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:32.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:32 smithi195 ceph-mon[33019]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T23:38:33.326 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T23:38:33.326 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T23:38:33.326 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-04-24T23:38:33.326 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-04-24T23:38:33.326 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00842567 s, 1.2 GB/s 2024-04-24T23:38:33.327 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-04-24T23:38:33.378 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:38:32.882+0000 7f356ed39700 -1 osd.3 0 waiting for initial osdmap 2024-04-24T23:38:33.378 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:38:32 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:38:32.889+0000 7f3566323700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:38:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:33 smithi170 ceph-mon[32714]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:38:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:33 smithi170 ceph-mon[32714]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T23:38:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:33 smithi170 ceph-mon[32714]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' 2024-04-24T23:38:33.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:33 smithi107 ceph-mon[29118]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:38:33.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:33 smithi107 ceph-mon[29118]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T23:38:33.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:33 smithi107 ceph-mon[29118]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' 2024-04-24T23:38:33.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:33 smithi195 ceph-mon[33019]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:38:33.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:33 smithi195 ceph-mon[33019]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T23:38:33.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:33.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:33 smithi195 ceph-mon[33019]: from='osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553]' entity='osd.3' 2024-04-24T23:38:33.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-04-24T23:38:34.202 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] boot 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:34 smithi170 ceph-mon[32714]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:34.770 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:38:34.770 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:38:34.770 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] boot 2024-04-24T23:38:34.770 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T23:38:34.771 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:34.771 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:34 smithi195 ceph-mon[33019]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: osd.3 [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] boot 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T23:38:34.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:34 smithi107 ceph-mon[29118]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi107 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi107 to 8544M 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:35.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:35.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi107 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi107 to 8544M 2024-04-24T23:38:35.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:35.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:35.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 -- 172.21.15.170:0/3522870622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 msgr2=0x7f27081015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 --2- 172.21.15.170:0/3522870622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f27081015b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f26f80099c0 tx=0x7f26f8009cd0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 -- 172.21.15.170:0/3522870622 shutdown_connections 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 --2- 172.21.15.170:0/3522870622 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 0x7f27081095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 --2- 172.21.15.170:0/3522870622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f27081015b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 --2- 172.21.15.170:0/3522870622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 0x7f27081027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 -- 172.21.15.170:0/3522870622 >> 172.21.15.170:0/3522870622 conn(0x7f27080fc730 msgr2=0x7f27080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 -- 172.21.15.170:0/3522870622 shutdown_connections 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 -- 172.21.15.170:0/3522870622 wait complete. 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.834+0000 7f270d516700 1 Processor -- start 2024-04-24T23:38:35.836 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 -- start start 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 0x7f270819d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 0x7f270819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f270810cd70 con 0x7f2708102390 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f270810cbf0 con 0x7f2708101190 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f270d516700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f270810cef0 con 0x7f2708102d30 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:37154/0 (socket says 172.21.15.170:37154) 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 -- 172.21.15.170:0/1834155440 learned_addr learned my addr 172.21.15.170:0/1834155440 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f26fe7fc700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 0x7f270819d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f27077fe700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 0x7f270819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 -- 172.21.15.170:0/1834155440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 msgr2=0x7f270819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 0x7f270819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 -- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 msgr2=0x7f270819d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 0x7f270819d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 -- 172.21.15.170:0/1834155440 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27081a00e0 con 0x7f2708101190 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.835+0000 7f2706ffd700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c010 tx=0x7f26f8009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.836+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8027070 con 0x7f2708101190 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.836+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27081a0370 con 0x7f2708101190 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.836+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26f8005940 con 0x7f2708101190 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.836+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8022dd0 con 0x7f2708101190 2024-04-24T23:38:35.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.836+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27081a08a0 con 0x7f2708101190 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.837+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2708068f20 con 0x7f2708101190 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.838+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f26f802a070 con 0x7f2708101190 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.839+0000 7f2704ff9700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 0x7f26f40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.839+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f26f80cf120 con 0x7f2708101190 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.839+0000 7f26fe7fc700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 0x7f26f40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:38:35.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.840+0000 7f26fe7fc700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 0x7f26f40a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f270806ba40 tx=0x7f26f00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:38:35.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:35.843+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f26f8031070 con 0x7f2708101190 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi107 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi107 to 8544M 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:35.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:38:35.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:36.049 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:36.049+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2708062730 con 0x7f26f40a2c20 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:36.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:36.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T23:38:36.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:36.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:36.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:36.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:36.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:38:36.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:37.680 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:37 smithi170 ceph-mon[32714]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T23:38:37.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:37 smithi195 ceph-mon[33019]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T23:38:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:37 smithi107 ceph-mon[29118]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T23:38:38.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:38 smithi170 ceph-mon[32714]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T23:38:38.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:38 smithi170 ceph-mon[32714]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T23:38:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:38 smithi195 ceph-mon[33019]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T23:38:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:38 smithi195 ceph-mon[33019]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T23:38:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:38 smithi107 ceph-mon[29118]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T23:38:38.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:38 smithi107 ceph-mon[29118]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T23:38:39.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:39 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/498231599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:39 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:39 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]': finished 2024-04-24T23:38:39.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:39 smithi195 ceph-mon[33019]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T23:38:39.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:39.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:39 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/498231599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:39 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:39 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]': finished 2024-04-24T23:38:39.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:39 smithi170 ceph-mon[32714]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T23:38:39.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:39.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:39 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/498231599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:39 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]: dispatch 2024-04-24T23:38:39.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:39 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7310213f-3a15-436a-a0bc-38c005a7aeb8"}]': finished 2024-04-24T23:38:39.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:39 smithi107 ceph-mon[29118]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T23:38:39.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/318336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:40 smithi195 ceph-mon[33019]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-24T23:38:40.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/318336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:40.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:40 smithi170 ceph-mon[32714]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-24T23:38:40.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/318336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:38:40.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:40 smithi107 ceph-mon[29118]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-24T23:38:43.155 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:42 smithi170 ceph-mon[32714]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:38:43.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:42 smithi195 ceph-mon[33019]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:38:43.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:42 smithi107 ceph-mon[29118]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:38:45.048 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:44 smithi170 ceph-mon[32714]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:44 smithi195 ceph-mon[33019]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:44 smithi107 ceph-mon[29118]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:46.988 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:46 smithi170 ceph-mon[32714]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:46.988 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T23:38:46.988 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:46.988 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:46 smithi170 ceph-mon[32714]: Deploying daemon osd.4 on smithi170 2024-04-24T23:38:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:46 smithi195 ceph-mon[33019]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T23:38:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:46 smithi195 ceph-mon[33019]: Deploying daemon osd.4 on smithi170 2024-04-24T23:38:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:46 smithi107 ceph-mon[29118]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T23:38:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:38:47.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:46 smithi107 ceph-mon[29118]: Deploying daemon osd.4 on smithi170 2024-04-24T23:38:49.240 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:48 smithi170 ceph-mon[32714]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:48 smithi195 ceph-mon[33019]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:48 smithi107 ceph-mon[29118]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:51.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:50 smithi170 ceph-mon[32714]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:50 smithi195 ceph-mon[33019]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:51.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:50 smithi107 ceph-mon[29118]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:52.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:52.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:52.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:52.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:52 smithi170 ceph-mon[32714]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:53.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:53.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:53.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:53.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:52 smithi195 ceph-mon[33019]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:38:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:53.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:52 smithi107 ceph-mon[29118]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:55.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:54 smithi170 ceph-mon[32714]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:55.162 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.162+0000 7f2704ff9700 1 -- 172.21.15.170:0/1834155440 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2708062730 con 0x7f26f40a2c20 2024-04-24T23:38:55.162 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 4 on host 'smithi170' 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 msgr2=0x7f26f40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 0x7f26f40a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f270806ba40 tx=0x7f26f00073d0 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 msgr2=0x7f270806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c010 tx=0x7f26f8009510 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 shutdown_connections 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f26f40a2c20 0x7f26f40a50e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2708102d30 0x7f270819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2708101190 0x7f270806b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 --2- 172.21.15.170:0/1834155440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2708102390 0x7f270819d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.164+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 >> 172.21.15.170:0/1834155440 conn(0x7f27080fc730 msgr2=0x7f2708105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:38:55.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.165+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 shutdown_connections 2024-04-24T23:38:55.166 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:38:55.165+0000 7f270d516700 1 -- 172.21.15.170:0/1834155440 wait complete. 2024-04-24T23:38:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:54 smithi195 ceph-mon[33019]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:55.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:54 smithi107 ceph-mon[29118]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:55.911 DEBUG:teuthology.orchestra.run.smithi170:osd.4> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.4.service 2024-04-24T23:38:55.913 INFO:tasks.cephadm:Deploying osd.5 on smithi195 with /dev/vg_nvme/lv_3... 2024-04-24T23:38:55.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T23:38:56.156 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.278 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T23:38:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:57.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-mon[32714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:57.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T23:38:57.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.428 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T23:38:57.429 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:57.429 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:57 smithi195 ceph-mon[33019]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.525 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4[43821]: 2024-04-24T23:38:57.158+0000 7f5a41988700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T23:38:57.526 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:38:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4[43821]: 2024-04-24T23:38:57.166+0000 7f5a3d77b700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:38:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T23:38:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: from='osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T23:38:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:57 smithi107 ceph-mon[29118]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:38:57.967 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T23:38:57.967 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T23:38:57.967 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-24T23:38:57.968 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-24T23:38:57.968 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00852264 s, 1.2 GB/s 2024-04-24T23:38:57.968 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-24T23:38:58.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:58 smithi195 ceph-mon[33019]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:38:58.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:58 smithi195 ceph-mon[33019]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T23:38:58.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.432 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:58 smithi170 ceph-mon[32714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:38:58.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:58 smithi170 ceph-mon[32714]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T23:38:58.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_3 2024-04-24T23:38:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:58 smithi107 ceph-mon[29118]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:38:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:58 smithi107 ceph-mon[29118]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T23:38:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:58.871 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:38:59.260 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:38:59.260 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:38:59.260 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:59.261 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:59.261 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] boot 2024-04-24T23:38:59.261 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T23:38:59.261 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:38:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:59.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:38:59.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:38:59.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:59.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:59.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] boot 2024-04-24T23:38:59.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T23:38:59.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:38:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: osd.4 [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] boot 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T23:38:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:38:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 -- 172.21.15.195:0/3711430715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102d30 msgr2=0x7f686c1095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 --2- 172.21.15.195:0/3711430715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102d30 0x7f686c1095e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6864009f90 tx=0x7f686400b330 comp rx=0 tx=0).stop 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 -- 172.21.15.195:0/3711430715 shutdown_connections 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 --2- 172.21.15.195:0/3711430715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102d30 0x7f686c1095e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 --2- 172.21.15.195:0/3711430715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 --2- 172.21.15.195:0/3711430715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102390 0x7f686c1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 -- 172.21.15.195:0/3711430715 >> 172.21.15.195:0/3711430715 conn(0x7f686c0fc730 msgr2=0x7f686c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 -- 172.21.15.195:0/3711430715 shutdown_connections 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.387+0000 7f68711a2700 1 -- 172.21.15.195:0/3711430715 wait complete. 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 Processor -- start 2024-04-24T23:39:00.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 -- start start 2024-04-24T23:39:00.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:00.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102d30 0x7f686c19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10cd80 con 0x7f686c102d30 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10cc00 con 0x7f686c101190 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f68711a2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10cf00 con 0x7f686c102390 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f686259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f686259c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:40892/0 (socket says 172.21.15.195:40892) 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f686259c700 1 -- 172.21.15.195:0/2030019681 learned_addr learned my addr 172.21.15.195:0/2030019681 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f686ad9d700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.388+0000 7f686b59e700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102d30 0x7f686c19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 -- 172.21.15.195:0/2030019681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 msgr2=0x7f686c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:00.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 -- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102d30 msgr2=0x7f686c19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102d30 0x7f686c19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 -- 172.21.15.195:0/2030019681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c1a00e0 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686259c700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f686ad9d700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c198480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f685c00e9c0 tx=0x7f685c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c004230 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c1a03d0 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f685c0043d0 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c010430 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.389+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c1a0910 con 0x7f686c101190 2024-04-24T23:39:00.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.390+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686c068f20 con 0x7f686c101190 2024-04-24T23:39:00.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.391+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f685c00c7c0 con 0x7f686c101190 2024-04-24T23:39:00.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.392+0000 7f6868d99700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 0x7f68580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:00.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.392+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f685c0ce3d0 con 0x7f686c101190 2024-04-24T23:39:00.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.392+0000 7f686259c700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 0x7f68580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:00.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.392+0000 7f686259c700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 0x7f68580a50e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6854005950 tx=0x7f6854009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:00.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.395+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f685c09c3a0 con 0x7f686c101190 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T23:39:00.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:00.597+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f686c0fe940 con 0x7f68580a2c20 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:00.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:00.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:00.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:01.528 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi170 2024-04-24T23:39:01.529 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi170 to 8544M 2024-04-24T23:39:01.529 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:01.529 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:01.529 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:01.529 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:01 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi170 2024-04-24T23:39:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi170 to 8544M 2024-04-24T23:39:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:01 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi170 2024-04-24T23:39:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi170 to 8544M 2024-04-24T23:39:01.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:01.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:01.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:01.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:01 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:02.400 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:02 smithi195 ceph-mon[33019]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:02.460 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:02 smithi107 ceph-mon[29118]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:02.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:02 smithi170 ceph-mon[32714]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:03.532 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:03 smithi195 ceph-mon[33019]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:03 smithi170 ceph-mon[32714]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:03.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:03 smithi107 ceph-mon[29118]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/2968341347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]': finished 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/136056471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:04.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/2968341347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]': finished 2024-04-24T23:39:04.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T23:39:04.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:04.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/136056471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:04.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/2968341347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]: dispatch 2024-04-24T23:39:04.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e45f9071-7234-40da-87e6-04473b1a0b07"}]': finished 2024-04-24T23:39:04.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T23:39:04.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:04.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/136056471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:05 smithi195 ceph-mon[33019]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:05 smithi170 ceph-mon[32714]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:05.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:05 smithi107 ceph-mon[29118]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:07.474 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:07 smithi195 ceph-mon[33019]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:07 smithi170 ceph-mon[32714]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:07.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:07 smithi107 ceph-mon[29118]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:09.301 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:09 smithi195 ceph-mon[33019]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:09.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:09 smithi170 ceph-mon[32714]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:09 smithi107 ceph-mon[29118]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:11.284 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:11 smithi195 ceph-mon[33019]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:11.284 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T23:39:11.284 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:11 smithi170 ceph-mon[32714]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T23:39:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:11.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:11 smithi107 ceph-mon[29118]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T23:39:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:12.280 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:12 smithi195 ceph-mon[33019]: Deploying daemon osd.5 on smithi195 2024-04-24T23:39:12.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:12 smithi170 ceph-mon[32714]: Deploying daemon osd.5 on smithi195 2024-04-24T23:39:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:12 smithi107 ceph-mon[29118]: Deploying daemon osd.5 on smithi195 2024-04-24T23:39:13.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:13 smithi195 ceph-mon[33019]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:13.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:13 smithi170 ceph-mon[32714]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:13.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:13 smithi107 ceph-mon[29118]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:15.394 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:15 smithi195 ceph-mon[33019]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:15 smithi170 ceph-mon[32714]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:15.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:15 smithi107 ceph-mon[29118]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:17.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:16 smithi195 ceph-mon[33019]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:17.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:16 smithi170 ceph-mon[32714]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:16 smithi107 ceph-mon[29118]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:19.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.811+0000 7f6868d99700 1 -- 172.21.15.195:0/2030019681 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f686c0fe940 con 0x7f68580a2c20 2024-04-24T23:39:19.812 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 5 on host 'smithi195' 2024-04-24T23:39:19.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 msgr2=0x7f68580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:19.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 0x7f68580a50e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6854005950 tx=0x7f6854009310 comp rx=0 tx=0).stop 2024-04-24T23:39:19.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 msgr2=0x7f686c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:19.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c198480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f685c00e9c0 tx=0x7f685c00ecd0 comp rx=0 tx=0).stop 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 shutdown_connections 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f68580a2c20 0x7f68580a50e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f686c102d30 0x7f686c19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f686c101190 0x7f686c198480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 --2- 172.21.15.195:0/2030019681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f686c102390 0x7f686c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 >> 172.21.15.195:0/2030019681 conn(0x7f686c0fc730 msgr2=0x7f686c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 shutdown_connections 2024-04-24T23:39:19.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:39:19.814+0000 7f68711a2700 1 -- 172.21.15.195:0/2030019681 wait complete. 2024-04-24T23:39:20.480 DEBUG:teuthology.orchestra.run.smithi195:osd.5> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.5.service 2024-04-24T23:39:20.482 INFO:tasks.cephadm:Deploying osd.6 on smithi107 with /dev/vg_nvme/lv_2... 2024-04-24T23:39:20.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T23:39:20.724 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:20 smithi107 ceph-mon[29118]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:20 smithi195 ceph-mon[33019]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:21.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:21.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:21.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T23:39:21.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:20 smithi170 ceph-mon[32714]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:21.958 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-mon[33019]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T23:39:21.958 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-mon[33019]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:21.958 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-mon[33019]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T23:39:21.958 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:21.958 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-mon[33019]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:21.958 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:39:21.811+0000 7f28e9ba1700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T23:39:21.959 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:39:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:39:21.817+0000 7f28e5994700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:39:22.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:21 smithi170 ceph-mon[32714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T23:39:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:21 smithi170 ceph-mon[32714]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:21 smithi170 ceph-mon[32714]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T23:39:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:21 smithi170 ceph-mon[32714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:22.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:21 smithi107 ceph-mon[29118]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T23:39:22.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:21 smithi107 ceph-mon[29118]: from='osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:22.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:21 smithi107 ceph-mon[29118]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T23:39:22.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:22.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:21 smithi107 ceph-mon[29118]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114947 s, 912 MB/s 2024-04-24T23:39:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] boot 2024-04-24T23:39:23.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T23:39:23.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:39:23.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T23:39:23.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:23.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] boot 2024-04-24T23:39:23.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T23:39:23.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: osd.5 [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] boot 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T23:39:23.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T23:39:23.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-04-24T23:39:23.666 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:39:24.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:23 smithi170 ceph-mon[32714]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:24.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:24.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:23 smithi107 ceph-mon[29118]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:24.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:24.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:24.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:24.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:23 smithi195 ceph-mon[33019]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T23:39:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:24 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi195 2024-04-24T23:39:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:24 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi195 to 8544M 2024-04-24T23:39:25.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:24 smithi107 ceph-mon[29118]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:25.023 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:24 smithi107 ceph-mon[29118]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T23:39:25.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:24 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi195 2024-04-24T23:39:25.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:24 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi195 to 8544M 2024-04-24T23:39:25.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:24 smithi170 ceph-mon[32714]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:25.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:24 smithi170 ceph-mon[32714]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T23:39:25.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 -- 172.21.15.107:0/2946013483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 msgr2=0x7f6980100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:25.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 --2- 172.21.15.107:0/2946013483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f6980100980 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6970009990 tx=0x7f6970009ca0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 -- 172.21.15.107:0/2946013483 shutdown_connections 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 --2- 172.21.15.107:0/2946013483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103ee0 0x7f698010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 --2- 172.21.15.107:0/2946013483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f6980100980 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 --2- 172.21.15.107:0/2946013483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103540 0x7f69801039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 -- 172.21.15.107:0/2946013483 >> 172.21.15.107:0/2946013483 conn(0x7f69800fbea0 msgr2=0x7f69800fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 -- 172.21.15.107:0/2946013483 shutdown_connections 2024-04-24T23:39:25.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.251+0000 7f6985029700 1 -- 172.21.15.107:0/2946013483 wait complete. 2024-04-24T23:39:25.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 Processor -- start 2024-04-24T23:39:25.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 -- start start 2024-04-24T23:39:25.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:25.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103540 0x7f6980198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:25.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103ee0 0x7f698019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f698010ded0 con 0x7f6980100560 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f698010dd50 con 0x7f6980103ee0 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f6985029700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f698010e050 con 0x7f6980103540 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.252+0000 7f697ed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56284/0 (socket says 172.21.15.107:56284) 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 -- 172.21.15.107:0/2701902153 learned_addr learned my addr 172.21.15.107:0/2701902153 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697f59e700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103ee0 0x7f698019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697e59c700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103540 0x7f6980198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 -- 172.21.15.107:0/2701902153 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103540 msgr2=0x7f6980198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103540 0x7f6980198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 -- 172.21.15.107:0/2701902153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103ee0 msgr2=0x7f698019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:25.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103ee0 0x7f698019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 -- 172.21.15.107:0/2701902153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69801a0150 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f697ed9d700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6970000c00 tx=0x7f69700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970003c70 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69801a03e0 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6970004830 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970005330 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.253+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69801a08b0 con 0x7f6980100560 2024-04-24T23:39:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.254+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6980068f20 con 0x7f6980100560 2024-04-24T23:39:25.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.255+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6970005550 con 0x7f6980100560 2024-04-24T23:39:25.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.256+0000 7f6967fff700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 0x7f69680a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:25.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.256+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f69700cf1f0 con 0x7f6980100560 2024-04-24T23:39:25.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.256+0000 7f697e59c700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 0x7f69680a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:25.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.256+0000 7f697e59c700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 0x7f69680a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6974005950 tx=0x7f69740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:25.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.260+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6970013070 con 0x7f6980100560 2024-04-24T23:39:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:24 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi195 2024-04-24T23:39:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:24 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi195 to 8544M 2024-04-24T23:39:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:24 smithi195 ceph-mon[33019]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:24 smithi195 ceph-mon[33019]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T23:39:25.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:25.466+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6980062670 con 0x7f69680a2c20 2024-04-24T23:39:26.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T23:39:26.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:26.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:26.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:26.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:26.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:26 smithi107 ceph-mon[29118]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:26.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T23:39:26.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:26.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:26.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:26.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:26.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:26 smithi195 ceph-mon[33019]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:26.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T23:39:26.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:26.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:26.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:26.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:26.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:26 smithi170 ceph-mon[32714]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:27.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:27 smithi107 ceph-mon[29118]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T23:39:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:27 smithi195 ceph-mon[33019]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T23:39:27.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:27 smithi170 ceph-mon[32714]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T23:39:28.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:28 smithi195 ceph-mon[33019]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-24T23:39:28.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:28 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]: dispatch 2024-04-24T23:39:28.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:28 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]': finished 2024-04-24T23:39:28.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:28 smithi195 ceph-mon[33019]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T23:39:28.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:28.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:28 smithi107 ceph-mon[29118]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-24T23:39:28.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:28 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]: dispatch 2024-04-24T23:39:28.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:28 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]': finished 2024-04-24T23:39:28.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:28 smithi107 ceph-mon[29118]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T23:39:28.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:28.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:28 smithi170 ceph-mon[32714]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-24T23:39:28.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:28 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]: dispatch 2024-04-24T23:39:28.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:28 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1625397055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1033c741-8643-41f0-b5e3-a7dfbb4d47d0"}]': finished 2024-04-24T23:39:28.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:28 smithi170 ceph-mon[32714]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T23:39:28.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:29.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:29 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1990274136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:29.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:29 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1990274136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:29.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:29 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1990274136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:30.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:30 smithi195 ceph-mon[33019]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-24T23:39:30.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:30 smithi170 ceph-mon[32714]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-24T23:39:30.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:30 smithi107 ceph-mon[29118]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-24T23:39:33.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:32 smithi107 ceph-mon[29118]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:39:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:32 smithi195 ceph-mon[33019]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:39:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:32 smithi170 ceph-mon[32714]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:39:35.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:34 smithi107 ceph-mon[29118]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:39:35.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:34 smithi195 ceph-mon[33019]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:39:35.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:34 smithi170 ceph-mon[32714]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:39:36.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T23:39:36.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:36.317 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T23:39:36.317 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:36.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T23:39:36.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:36 smithi107 ceph-mon[29118]: Deploying daemon osd.6 on smithi107 2024-04-24T23:39:37.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:36 smithi107 ceph-mon[29118]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:36 smithi195 ceph-mon[33019]: Deploying daemon osd.6 on smithi107 2024-04-24T23:39:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:36 smithi195 ceph-mon[33019]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:36 smithi170 ceph-mon[32714]: Deploying daemon osd.6 on smithi107 2024-04-24T23:39:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:36 smithi170 ceph-mon[32714]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:39.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:38 smithi195 ceph-mon[33019]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:39.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:38 smithi107 ceph-mon[29118]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:39.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:38 smithi170 ceph-mon[32714]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:41.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:41 smithi195 ceph-mon[33019]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:41 smithi170 ceph-mon[32714]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:41.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:41 smithi107 ceph-mon[29118]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:42.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:42.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:42 smithi107 ceph-mon[29118]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:42 smithi195 ceph-mon[33019]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:42 smithi170 ceph-mon[32714]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:45.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:45.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:44 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:44 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:45.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:45.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:45.349 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:44 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:45.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.422+0000 7f6967fff700 1 -- 172.21.15.107:0/2701902153 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6980062670 con 0x7f69680a2c20 2024-04-24T23:39:45.427 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 6 on host 'smithi107' 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 msgr2=0x7f69680a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 0x7f69680a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6974005950 tx=0x7f69740058e0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 msgr2=0x7f69801984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6970000c00 tx=0x7f69700047b0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 shutdown_connections 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f69680a2c20 0x7f69680a50e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6980103ee0 0x7f698019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6980100560 0x7f69801984f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 --2- 172.21.15.107:0/2701902153 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6980103540 0x7f6980198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 >> 172.21.15.107:0/2701902153 conn(0x7f69800fbea0 msgr2=0x7f6980106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:39:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 shutdown_connections 2024-04-24T23:39:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:39:45.424+0000 7f6985029700 1 -- 172.21.15.107:0/2701902153 wait complete. 2024-04-24T23:39:46.036 DEBUG:teuthology.orchestra.run.smithi107:osd.6> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.6.service 2024-04-24T23:39:46.113 INFO:tasks.cephadm:Deploying osd.7 on smithi170 with /dev/vg_nvme/lv_2... 2024-04-24T23:39:46.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T23:39:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:45 smithi195 ceph-mon[33019]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T23:39:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:46.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:45 smithi107 ceph-mon[29118]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T23:39:46.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:45 smithi170 ceph-mon[32714]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T23:39:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:39:46.357 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:39:47.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:47 smithi170 ceph-mon[32714]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:47.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:47 smithi170 ceph-mon[32714]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T23:39:47.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:47 smithi170 ceph-mon[32714]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T23:39:47.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:47 smithi170 ceph-mon[32714]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:39:47.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:47 smithi195 ceph-mon[33019]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:47 smithi195 ceph-mon[33019]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T23:39:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:47 smithi195 ceph-mon[33019]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T23:39:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:47 smithi195 ceph-mon[33019]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:39:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:47.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-mon[29118]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:47.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-mon[29118]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T23:39:47.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-mon[29118]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T23:39:47.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-mon[29118]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:39:47.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:47.350 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:39:47.002+0000 7fb76aea9700 -1 osd.6 0 waiting for initial osdmap 2024-04-24T23:39:47.350 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:39:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:39:47.010+0000 7fb766c9c700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:39:48.116 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:48 smithi170 ceph-mon[32714]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:39:48.116 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:48 smithi170 ceph-mon[32714]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T23:39:48.117 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.117 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:48 smithi107 ceph-mon[29118]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:39:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:48 smithi107 ceph-mon[29118]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T23:39:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:48 smithi195 ceph-mon[33019]: from='osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:39:48.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:48 smithi195 ceph-mon[33019]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T23:39:48.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:48.573 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T23:39:48.573 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T23:39:48.574 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-04-24T23:39:48.574 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-04-24T23:39:48.574 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00857088 s, 1.2 GB/s 2024-04-24T23:39:48.574 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-04-24T23:39:49.229 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] boot 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:49.231 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:49.231 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] boot 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: osd.6 [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] boot 2024-04-24T23:39:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:39:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:39:49.494 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:39:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:50 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi107 2024-04-24T23:39:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:50 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi107 to 5696M 2024-04-24T23:39:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:50 smithi195 ceph-mon[33019]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T23:39:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:50 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi107 2024-04-24T23:39:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:50 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi107 to 5696M 2024-04-24T23:39:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:50 smithi170 ceph-mon[32714]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T23:39:50.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:50 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi107 2024-04-24T23:39:50.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:50 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi107 to 5696M 2024-04-24T23:39:50.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:50 smithi107 ceph-mon[29118]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 -- 172.21.15.170:0/1986628422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c1024c0 msgr2=0x7f389c108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1986628422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c1024c0 0x7f389c108d70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f389400b0a0 tx=0x7f389400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 -- 172.21.15.170:0/1986628422 shutdown_connections 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1986628422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c1024c0 0x7f389c108d70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1986628422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 0x7f389c100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1986628422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c101b20 0x7f389c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 -- 172.21.15.170:0/1986628422 >> 172.21.15.170:0/1986628422 conn(0x7f389c0fbea0 msgr2=0x7f389c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 -- 172.21.15.170:0/1986628422 shutdown_connections 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.254+0000 7f38a28fa700 1 -- 172.21.15.170:0/1986628422 wait complete. 2024-04-24T23:39:51.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 Processor -- start 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 -- start start 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 0x7f389c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c1024c0 0x7f389c19f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f389c10df30 con 0x7f389c1024c0 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f389c10ddb0 con 0x7f389c101b20 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f38a28fa700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f389c10e0b0 con 0x7f389c100920 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f389b7fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.255+0000 7f389b7fe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33534/0 (socket says 172.21.15.170:33534) 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 -- 172.21.15.170:0/1283944359 learned_addr learned my addr 172.21.15.170:0/1283944359 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38a0e97700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c1024c0 0x7f389c19f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389bfff700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 0x7f389c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:51.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 -- 172.21.15.170:0/1283944359 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 msgr2=0x7f389c1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 0x7f389c1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 -- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c1024c0 msgr2=0x7f389c19f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c1024c0 0x7f389c19f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 -- 172.21.15.170:0/1283944359 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389c1a0020 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f389b7fe700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f389000d950 tx=0x7f389000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890016070 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3890009e20 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389c1a0310 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389000f620 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.256+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389c1a0850 con 0x7f389c101b20 2024-04-24T23:39:51.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.257+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389c068f20 con 0x7f389c101b20 2024-04-24T23:39:51.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.258+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3890019070 con 0x7f389c101b20 2024-04-24T23:39:51.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.259+0000 7f38997fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 0x7f38840a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:39:51.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.259+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f38900cdf00 con 0x7f389c101b20 2024-04-24T23:39:51.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.259+0000 7f389bfff700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 0x7f38840a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:39:51.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.259+0000 7f389bfff700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 0x7f38840a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f388c005950 tx=0x7f388c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:39:51.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.262+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f389009bb70 con 0x7f389c101b20 2024-04-24T23:39:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:51 smithi195 ceph-mon[33019]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:51 smithi170 ceph-mon[32714]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:51.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:51 smithi107 ceph-mon[29118]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:51.470 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:39:51.470+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f389c062730 con 0x7f38840a2c20 2024-04-24T23:39:52.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:52.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:52.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:52 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:52 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:39:52.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:39:52.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:52 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:39:53.295 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:53 smithi170 ceph-mon[32714]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:53.295 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:53 smithi170 ceph-mon[32714]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:53 smithi195 ceph-mon[33019]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:53 smithi195 ceph-mon[33019]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:53.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:53 smithi107 ceph-mon[29118]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:39:53.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:53 smithi107 ceph-mon[29118]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/812819826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]': finished 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T23:39:55.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:39:55.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:55 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/658460466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/812819826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]': finished 2024-04-24T23:39:55.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T23:39:55.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:39:55.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:55 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/658460466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/812819826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]: dispatch 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42a4f20d-28a4-4910-9850-ebd9c9919974"}]': finished 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:39:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:55 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/658460466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:39:57.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:57 smithi195 ceph-mon[33019]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:57.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:57 smithi170 ceph-mon[32714]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:57.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:57 smithi107 ceph-mon[29118]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:59.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:39:59 smithi170 ceph-mon[32714]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:59.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:39:59 smithi195 ceph-mon[33019]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:39:59.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:39:59 smithi107 ceph-mon[29118]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:00.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:00 smithi195 ceph-mon[33019]: overall HEALTH_OK 2024-04-24T23:40:00.331 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:00 smithi170 ceph-mon[32714]: overall HEALTH_OK 2024-04-24T23:40:00.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:00 smithi107 ceph-mon[29118]: overall HEALTH_OK 2024-04-24T23:40:01.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:01 smithi195 ceph-mon[33019]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:01.329 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:01 smithi170 ceph-mon[32714]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:01.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:01 smithi107 ceph-mon[29118]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:02.317 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T23:40:02.317 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:02 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:02.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T23:40:02.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:02 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:02.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T23:40:02.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:02 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:03.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:03 smithi170 ceph-mon[32714]: Deploying daemon osd.7 on smithi170 2024-04-24T23:40:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:03 smithi170 ceph-mon[32714]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:03.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:03 smithi107 ceph-mon[29118]: Deploying daemon osd.7 on smithi170 2024-04-24T23:40:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:03 smithi107 ceph-mon[29118]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:03 smithi195 ceph-mon[33019]: Deploying daemon osd.7 on smithi170 2024-04-24T23:40:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:03 smithi195 ceph-mon[33019]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:05.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:05 smithi170 ceph-mon[32714]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:05.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:05 smithi107 ceph-mon[29118]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:05 smithi195 ceph-mon[33019]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:07.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:07 smithi170 ceph-mon[32714]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:07.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:07 smithi107 ceph-mon[29118]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:07 smithi195 ceph-mon[33019]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:08.154 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:08.154 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:08.154 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:08.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:08.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:08.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:09.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:09 smithi170 ceph-mon[32714]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:09.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:09 smithi107 ceph-mon[29118]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:09 smithi195 ceph-mon[33019]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:11.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:11.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:11.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:11.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.384 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 7 on host 'smithi170' 2024-04-24T23:40:11.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.384+0000 7f38997fa700 1 -- 172.21.15.170:0/1283944359 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f389c062730 con 0x7f38840a2c20 2024-04-24T23:40:11.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 msgr2=0x7f38840a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:11.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 0x7f38840a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f388c005950 tx=0x7f388c0058e0 comp rx=0 tx=0).stop 2024-04-24T23:40:11.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 msgr2=0x7f389c198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:11.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f389000d950 tx=0x7f389000dc60 comp rx=0 tx=0).stop 2024-04-24T23:40:11.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 shutdown_connections 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38840a2c20 0x7f38840a50e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f389c1024c0 0x7f389c19f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f389c100920 0x7f389c1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 --2- 172.21.15.170:0/1283944359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f389c101b20 0x7f389c198900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 >> 172.21.15.170:0/1283944359 conn(0x7f389c0fbea0 msgr2=0x7f389c104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 shutdown_connections 2024-04-24T23:40:11.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:40:11.386+0000 7f38a28fa700 1 -- 172.21.15.170:0/1283944359 wait complete. 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.060 DEBUG:teuthology.orchestra.run.smithi170:osd.7> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.7.service 2024-04-24T23:40:12.062 INFO:tasks.cephadm:Deploying osd.8 on smithi195 with /dev/vg_nvme/lv_2... 2024-04-24T23:40:12.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T23:40:12.308 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:12.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:12 smithi107 ceph-mon[29118]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:12.668 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:12 smithi195 ceph-mon[33019]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:12.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:12 smithi170 ceph-mon[32714]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:13.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-mon[32714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T23:40:13.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-mon[32714]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:13.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-mon[32714]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T23:40:13.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:13.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-mon[32714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:13.700 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:13 smithi195 ceph-mon[33019]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T23:40:13.701 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:13 smithi195 ceph-mon[33019]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:13.701 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:13 smithi195 ceph-mon[33019]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T23:40:13.701 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:13 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:13.701 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:13 smithi195 ceph-mon[33019]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:13 smithi107 ceph-mon[29118]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T23:40:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:13 smithi107 ceph-mon[29118]: from='osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:13 smithi107 ceph-mon[29118]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T23:40:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:13 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:13 smithi107 ceph-mon[29118]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:40:14.085 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:40:13.821+0000 7fde21a80700 -1 osd.7 0 waiting for initial osdmap 2024-04-24T23:40:14.086 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:40:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:40:13.829+0000 7fde1986b700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:40:14.467 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T23:40:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T23:40:14.468 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-24T23:40:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-24T23:40:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850043 s, 1.2 GB/s 2024-04-24T23:40:14.468 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: from='osd.7 ' entity='osd.7' 2024-04-24T23:40:14.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:14 smithi195 ceph-mon[33019]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:14.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: from='osd.7 ' entity='osd.7' 2024-04-24T23:40:14.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:14 smithi170 ceph-mon[32714]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: from='osd.7 ' entity='osd.7' 2024-04-24T23:40:14.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:14 smithi107 ceph-mon[29118]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-24T23:40:15.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_2 2024-04-24T23:40:15.363 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:40:15.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:15 smithi170 ceph-mon[32714]: osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] boot 2024-04-24T23:40:15.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:15 smithi170 ceph-mon[32714]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T23:40:15.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:15 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:15.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:15 smithi170 ceph-mon[32714]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T23:40:15.700 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:15 smithi195 ceph-mon[33019]: osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] boot 2024-04-24T23:40:15.700 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:15 smithi195 ceph-mon[33019]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T23:40:15.700 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:15 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:15.700 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:15 smithi195 ceph-mon[33019]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T23:40:15.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:15 smithi107 ceph-mon[29118]: osd.7 [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] boot 2024-04-24T23:40:15.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:15 smithi107 ceph-mon[29118]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T23:40:15.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:15 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T23:40:15.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:15 smithi107 ceph-mon[29118]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T23:40:16.684 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi170 2024-04-24T23:40:16.684 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi170 to 5696M 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:16.685 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:16 smithi195 ceph-mon[33019]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi170 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi170 to 5696M 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:16.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:16.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:16 smithi170 ceph-mon[32714]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi170 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi170 to 5696M 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:16.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:16.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:16.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:16 smithi107 ceph-mon[29118]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T23:40:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 -- 172.21.15.195:0/992757167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660103e30 msgr2=0x7ff66010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 --2- 172.21.15.195:0/992757167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660103e30 0x7ff66010a6e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff65800b0a0 tx=0x7ff65800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 -- 172.21.15.195:0/992757167 shutdown_connections 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 --2- 172.21.15.195:0/992757167 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660103e30 0x7ff66010a6e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 --2- 172.21.15.195:0/992757167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660102290 0x7ff6601026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 --2- 172.21.15.195:0/992757167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103490 0x7ff6601038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 -- 172.21.15.195:0/992757167 >> 172.21.15.195:0/992757167 conn(0x7ff6600fd830 msgr2=0x7ff6600ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 -- 172.21.15.195:0/992757167 shutdown_connections 2024-04-24T23:40:16.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.894+0000 7ff6656b3700 1 -- 172.21.15.195:0/992757167 wait complete. 2024-04-24T23:40:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 Processor -- start 2024-04-24T23:40:16.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 -- start start 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 0x7ff6601989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103e30 0x7ff66019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010ded0 con 0x7ff660103490 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010dd50 con 0x7ff660103e30 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff6656b3700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010e050 con 0x7ff660102290 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:57344/0 (socket says 172.21.15.195:57344) 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 -- 172.21.15.195:0/3122417383 learned_addr learned my addr 172.21.15.195:0/3122417383 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65e7fc700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 0x7ff6601989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65f7fe700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103e30 0x7ff66019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:16.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 -- 172.21.15.195:0/3122417383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103e30 msgr2=0x7ff66019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103e30 0x7ff66019fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 -- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 msgr2=0x7ff6601989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 0x7ff6601989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65effd700 1 -- 172.21.15.195:0/3122417383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6601a00e0 con 0x7ff660102290 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.895+0000 7ff65e7fc700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 0x7ff6601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff65effd700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff65000d9b0 tx=0x7ff65000dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650016070 con 0x7ff660102290 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff65000bb60 con 0x7ff660102290 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6601a03d0 con 0x7ff660102290 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65000f3c0 con 0x7ff660102290 2024-04-24T23:40:16.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.896+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6601a0910 con 0x7ff660102290 2024-04-24T23:40:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.897+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff660068f20 con 0x7ff660102290 2024-04-24T23:40:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.898+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff650019070 con 0x7ff660102290 2024-04-24T23:40:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.898+0000 7ff647fff700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 0x7ff6480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.899+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ff6500ce9a0 con 0x7ff660102290 2024-04-24T23:40:16.899 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.899+0000 7ff65e7fc700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 0x7ff6480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:16.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.899+0000 7ff65e7fc700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 0x7ff6480a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff654007900 tx=0x7ff654008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:40:16.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:16.902+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff650020030 con 0x7ff660102290 2024-04-24T23:40:17.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:17.107+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff660062670 con 0x7ff6480a2c20 2024-04-24T23:40:17.875 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:17 smithi195 ceph-mon[33019]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:17.875 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:17 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:17.875 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:17 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:17.875 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:17 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:17.875 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:17 smithi195 ceph-mon[33019]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T23:40:17.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:17 smithi107 ceph-mon[29118]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:17.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:17 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:17.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:17 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:17.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:17 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:17.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:17 smithi107 ceph-mon[29118]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T23:40:18.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:17 smithi170 ceph-mon[32714]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:18.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:17 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:18.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:17 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:18.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:17 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:18.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:17 smithi170 ceph-mon[32714]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T23:40:18.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:18 smithi107 ceph-mon[29118]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:18.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:18 smithi107 ceph-mon[29118]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T23:40:18.984 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:18 smithi195 ceph-mon[33019]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:18.984 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:18 smithi195 ceph-mon[33019]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T23:40:19.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:18 smithi170 ceph-mon[32714]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:19.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:18 smithi170 ceph-mon[32714]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T23:40:20.191 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/206941608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.191 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:19 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.191 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:19 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]': finished 2024-04-24T23:40:20.191 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:19 smithi195 ceph-mon[33019]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T23:40:20.191 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/206941608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:19 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:19 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]': finished 2024-04-24T23:40:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:19 smithi170 ceph-mon[32714]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T23:40:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/206941608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:19 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]: dispatch 2024-04-24T23:40:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:19 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9844dda-7739-49ea-9548-685903258f80"}]': finished 2024-04-24T23:40:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:19 smithi107 ceph-mon[29118]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T23:40:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:20 smithi107 ceph-mon[29118]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:20 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/2316463973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:21.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:20 smithi195 ceph-mon[33019]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:21.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:20 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/2316463973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:21.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:20 smithi170 ceph-mon[32714]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:21.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:20 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/2316463973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:22 smithi195 ceph-mon[33019]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:23.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:22 smithi170 ceph-mon[32714]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:23.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:22 smithi107 ceph-mon[29118]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:25.295 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:24 smithi195 ceph-mon[33019]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:25.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:24 smithi170 ceph-mon[32714]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:25.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:24 smithi107 ceph-mon[29118]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:27.096 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:27 smithi195 ceph-mon[33019]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:27 smithi170 ceph-mon[32714]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:27.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:27 smithi107 ceph-mon[29118]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:28.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T23:40:28.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:28.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:28 smithi195 ceph-mon[33019]: Deploying daemon osd.8 on smithi195 2024-04-24T23:40:28.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T23:40:28.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:28.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:28 smithi170 ceph-mon[32714]: Deploying daemon osd.8 on smithi195 2024-04-24T23:40:28.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T23:40:28.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:28.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:28 smithi107 ceph-mon[29118]: Deploying daemon osd.8 on smithi195 2024-04-24T23:40:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:29 smithi195 ceph-mon[33019]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:29 smithi170 ceph-mon[32714]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:29.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:29 smithi107 ceph-mon[29118]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:31.049 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:31 smithi195 ceph-mon[33019]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:31.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:31 smithi170 ceph-mon[32714]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:31.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:31 smithi107 ceph-mon[29118]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:33 smithi195 ceph-mon[33019]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:33 smithi170 ceph-mon[32714]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:33.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:33 smithi107 ceph-mon[29118]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:33.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:33.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:35.169 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:35 smithi195 ceph-mon[33019]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:35.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:35 smithi170 ceph-mon[32714]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:35.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:35 smithi107 ceph-mon[29118]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:36.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.075+0000 7ff647fff700 1 -- 172.21.15.195:0/3122417383 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff660062670 con 0x7ff6480a2c20 2024-04-24T23:40:36.075 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 8 on host 'smithi195' 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 msgr2=0x7ff6480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 0x7ff6480a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff654007900 tx=0x7ff654008040 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 msgr2=0x7ff660198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff65000d9b0 tx=0x7ff65000dd70 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 shutdown_connections 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6480a2c20 0x7ff6480a50e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff660103e30 0x7ff66019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff660102290 0x7ff660198480 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 --2- 172.21.15.195:0/3122417383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff660103490 0x7ff6601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:36.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 >> 172.21.15.195:0/3122417383 conn(0x7ff6600fd830 msgr2=0x7ff660106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:40:36.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 shutdown_connections 2024-04-24T23:40:36.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:40:36.077+0000 7ff6656b3700 1 -- 172.21.15.195:0/3122417383 wait complete. 2024-04-24T23:40:37.061 DEBUG:teuthology.orchestra.run.smithi195:osd.8> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.8.service 2024-04-24T23:40:37.064 INFO:tasks.cephadm:Deploying osd.9 on smithi107 with /dev/vg_nvme/lv_1... 2024-04-24T23:40:37.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T23:40:37.311 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:40:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:37 smithi195 ceph-mon[33019]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:37 smithi195 ceph-mon[33019]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:37 smithi195 ceph-mon[33019]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:37 smithi170 ceph-mon[32714]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:37 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:37 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:37 smithi170 ceph-mon[32714]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:37 smithi170 ceph-mon[32714]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:37.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:37 smithi107 ceph-mon[29118]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:37.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:37 smithi107 ceph-mon[29118]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:37 smithi107 ceph-mon[29118]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T23:40:38.109 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T23:40:38.109 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.109 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T23:40:38.110 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:38.110 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.110 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.110 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:38 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:38 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:38.812 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:40:38.511+0000 7f2ab6a50700 -1 osd.8 0 waiting for initial osdmap 2024-04-24T23:40:38.812 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:40:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:40:38.522+0000 7f2aae83b700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:40:39.097 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:39.097 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:40:39.097 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T23:40:39.097 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.097 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.098 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:39 smithi195 ceph-mon[33019]: from='osd.8 ' entity='osd.8' 2024-04-24T23:40:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T23:40:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T23:40:39.208 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-04-24T23:40:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-04-24T23:40:39.208 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120332 s, 871 MB/s 2024-04-24T23:40:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-04-24T23:40:39.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:39.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:40:39.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T23:40:39.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:39 smithi170 ceph-mon[32714]: from='osd.8 ' entity='osd.8' 2024-04-24T23:40:39.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-24T23:40:39.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:40:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T23:40:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:39.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:39 smithi107 ceph-mon[29118]: from='osd.8 ' entity='osd.8' 2024-04-24T23:40:39.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] boot 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: osdmap e59: 9 total, 9 up, 9 in 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:40.229 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:40 smithi195 ceph-mon[33019]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T23:40:40.232 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:40:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:40:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:40:40.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] boot 2024-04-24T23:40:40.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: osdmap e59: 9 total, 9 up, 9 in 2024-04-24T23:40:40.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:40.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:40 smithi170 ceph-mon[32714]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: osd.8 [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] boot 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: osdmap e59: 9 total, 9 up, 9 in 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T23:40:40.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:40 smithi107 ceph-mon[29118]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T23:40:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:41 smithi170 ceph-mon[32714]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T23:40:41.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:41.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:41 smithi107 ceph-mon[29118]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:41.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:41 smithi195 ceph-mon[33019]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T23:40:42.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 -- 172.21.15.107:0/2667711363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c105640 msgr2=0x7fad9c107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 --2- 172.21.15.107:0/2667711363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c105640 0x7fad9c107a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009990 tx=0x7fad8c009ca0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 -- 172.21.15.107:0/2667711363 shutdown_connections 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 --2- 172.21.15.107:0/2667711363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 0x7fad9c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 --2- 172.21.15.107:0/2667711363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c102d10 0x7fad9c105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.034+0000 7fada32c2700 1 --2- 172.21.15.107:0/2667711363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c105640 0x7fad9c107a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- 172.21.15.107:0/2667711363 >> 172.21.15.107:0/2667711363 conn(0x7fad9c0fc720 msgr2=0x7fad9c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- 172.21.15.107:0/2667711363 shutdown_connections 2024-04-24T23:40:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- 172.21.15.107:0/2667711363 wait complete. 2024-04-24T23:40:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 Processor -- start 2024-04-24T23:40:42.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- start start 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 0x7fad9c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 0x7fad9c1a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10ded0 con 0x7fad9c102d10 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10dd50 con 0x7fad9c105640 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada32c2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10e050 con 0x7fad9c107f70 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada105e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada105e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51062/0 (socket says 172.21.15.107:51062) 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.035+0000 7fada105e700 1 -- 172.21.15.107:0/4022840511 learned_addr learned my addr 172.21.15.107:0/4022840511 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada085d700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 0x7fad9c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada185f700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 0x7fad9c1a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 -- 172.21.15.107:0/4022840511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 msgr2=0x7fad9c1a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:42.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 0x7fad9c1a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 -- 172.21.15.107:0/4022840511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 msgr2=0x7fad9c19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 0x7fad9c19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 -- 172.21.15.107:0/4022840511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9c1a2330 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada085d700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 0x7fad9c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada105e700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c950 tx=0x7fad9800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad98007ac0 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad98004d10 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9800f450 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9c1a2620 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.036+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9c1a2c30 con 0x7fad9c102d10 2024-04-24T23:40:42.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.037+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad9c068f20 con 0x7fad9c102d10 2024-04-24T23:40:42.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.038+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fad98004750 con 0x7fad9c102d10 2024-04-24T23:40:42.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.039+0000 7fad927fc700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 0x7fad880a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:40:42.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.039+0000 7fada085d700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 0x7fad880a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:40:42.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.039+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fad980ce600 con 0x7fad9c102d10 2024-04-24T23:40:42.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.039+0000 7fada085d700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 0x7fad880a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b5c0 tx=0x7fad8c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:40:42.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.042+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad98014030 con 0x7fad9c102d10 2024-04-24T23:40:42.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:40:42.251+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fad9c062670 con 0x7fad880a2ca0 2024-04-24T23:40:42.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:42 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi195 2024-04-24T23:40:42.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:42 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi195 to 5696M 2024-04-24T23:40:42.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:42 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi195 2024-04-24T23:40:42.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:42 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi195 to 5696M 2024-04-24T23:40:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:42 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi195 2024-04-24T23:40:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:42 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi195 to 5696M 2024-04-24T23:40:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:43 smithi107 ceph-mon[29118]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:43 smithi107 ceph-mon[29118]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T23:40:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:43 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:43 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:43 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:43.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:43 smithi170 ceph-mon[32714]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:43.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:43 smithi170 ceph-mon[32714]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T23:40:43.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:43 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:43.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:43 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:43.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:43 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:43 smithi195 ceph-mon[33019]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:43 smithi195 ceph-mon[33019]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T23:40:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:43 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:40:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:43 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:40:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:43 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:44.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:44 smithi107 ceph-mon[29118]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:44.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:44 smithi107 ceph-mon[29118]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T23:40:44.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:44 smithi170 ceph-mon[32714]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:44.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:44 smithi170 ceph-mon[32714]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T23:40:44.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:44 smithi195 ceph-mon[33019]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:40:44.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:44 smithi195 ceph-mon[33019]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T23:40:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:45 smithi170 ceph-mon[32714]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:45.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:45 smithi107 ceph-mon[29118]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:45 smithi195 ceph-mon[33019]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]: dispatch 2024-04-24T23:40:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]': finished 2024-04-24T23:40:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:46 smithi170 ceph-mon[32714]: osdmap e64: 10 total, 9 up, 10 in 2024-04-24T23:40:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:40:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2607386760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]: dispatch 2024-04-24T23:40:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]': finished 2024-04-24T23:40:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:46 smithi107 ceph-mon[29118]: osdmap e64: 10 total, 9 up, 10 in 2024-04-24T23:40:46.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:40:46.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2607386760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]: dispatch 2024-04-24T23:40:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3389569462' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98528d16-0e91-4f6c-99ed-71e534aa6551"}]': finished 2024-04-24T23:40:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:46 smithi195 ceph-mon[33019]: osdmap e64: 10 total, 9 up, 10 in 2024-04-24T23:40:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:40:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2607386760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:40:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:47 smithi195 ceph-mon[33019]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-24T23:40:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:47 smithi170 ceph-mon[32714]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-24T23:40:47.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:47 smithi107 ceph-mon[29118]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-24T23:40:49.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:49 smithi107 ceph-mon[29118]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:40:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:49 smithi195 ceph-mon[33019]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:40:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:49 smithi170 ceph-mon[32714]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T23:40:51.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:51 smithi107 ceph-mon[29118]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-24T23:40:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:51 smithi195 ceph-mon[33019]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-24T23:40:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:51 smithi170 ceph-mon[32714]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-24T23:40:53.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:53 smithi107 ceph-mon[29118]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T23:40:53.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:53 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T23:40:53.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:53 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:53 smithi195 ceph-mon[33019]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T23:40:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:53 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T23:40:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:53 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:53 smithi170 ceph-mon[32714]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T23:40:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:53 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T23:40:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:53 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:40:54.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:54 smithi107 ceph-mon[29118]: Deploying daemon osd.9 on smithi107 2024-04-24T23:40:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:54 smithi195 ceph-mon[33019]: Deploying daemon osd.9 on smithi107 2024-04-24T23:40:54.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:54 smithi170 ceph-mon[32714]: Deploying daemon osd.9 on smithi107 2024-04-24T23:40:55.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:55 smithi107 ceph-mon[29118]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T23:40:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:55 smithi195 ceph-mon[33019]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T23:40:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:55 smithi170 ceph-mon[32714]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T23:40:57.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:57 smithi107 ceph-mon[29118]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:57 smithi195 ceph-mon[33019]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:57.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:57 smithi170 ceph-mon[32714]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:59.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:59 smithi107 ceph-mon[29118]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:59.462 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.462 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.462 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:40:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:59 smithi195 ceph-mon[33019]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:40:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:40:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:59 smithi170 ceph-mon[32714]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:40:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:40:59.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:40:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:01.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:01 smithi107 ceph-mon[29118]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:01 smithi195 ceph-mon[33019]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:01 smithi170 ceph-mon[32714]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:02.056 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 9 on host 'smithi107' 2024-04-24T23:41:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.055+0000 7fad927fc700 1 -- 172.21.15.107:0/4022840511 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fad9c062670 con 0x7fad880a2ca0 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 msgr2=0x7fad880a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 0x7fad880a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b5c0 tx=0x7fad8c005fb0 comp rx=0 tx=0).stop 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 msgr2=0x7fad9c19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c950 tx=0x7fad9800cd10 comp rx=0 tx=0).stop 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 shutdown_connections 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fad880a2ca0 0x7fad880a5160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fad9c107f70 0x7fad9c1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad9c102d10 0x7fad9c19a6d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.057+0000 7fada32c2700 1 --2- 172.21.15.107:0/4022840511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c105640 0x7fad9c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.058+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 >> 172.21.15.107:0/4022840511 conn(0x7fad9c0fc720 msgr2=0x7fad9c1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.058+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 shutdown_connections 2024-04-24T23:41:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:02.058+0000 7fada32c2700 1 -- 172.21.15.107:0/4022840511 wait complete. 2024-04-24T23:41:02.736 DEBUG:teuthology.orchestra.run.smithi107:osd.9> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.9.service 2024-04-24T23:41:02.738 INFO:tasks.cephadm:Deploying osd.10 on smithi170 with /dev/vg_nvme/lv_1... 2024-04-24T23:41:02.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T23:41:02.987 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:03 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:03 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T23:41:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:03.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:03 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:04.439 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:04 smithi170 ceph-mon[32714]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T23:41:04.440 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:04 smithi170 ceph-mon[32714]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T23:41:04.440 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:04 smithi170 ceph-mon[32714]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:41:04.440 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:04 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:04.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-mon[29118]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T23:41:04.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-mon[29118]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T23:41:04.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-mon[29118]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:41:04.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:04.513 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:41:04.168+0000 7f4512f6d700 -1 osd.9 0 waiting for initial osdmap 2024-04-24T23:41:04.513 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:41:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:41:04.184+0000 7f450ed60700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:41:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:04 smithi195 ceph-mon[33019]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T23:41:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:04 smithi195 ceph-mon[33019]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T23:41:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:04 smithi195 ceph-mon[33019]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-04-24T23:41:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:04 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.019 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T23:41:05.020 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T23:41:05.020 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-04-24T23:41:05.020 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-04-24T23:41:05.020 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850138 s, 1.2 GB/s 2024-04-24T23:41:05.020 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-04-24T23:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:05 smithi107 ceph-mon[29118]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:05 smithi107 ceph-mon[29118]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:05 smithi107 ceph-mon[29118]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T23:41:05.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:05 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:05 smithi170 ceph-mon[32714]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:05.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:05 smithi170 ceph-mon[32714]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:41:05.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:05 smithi170 ceph-mon[32714]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T23:41:05.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.497 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:05 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:05 smithi195 ceph-mon[33019]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:05 smithi195 ceph-mon[33019]: from='osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-04-24T23:41:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:05 smithi195 ceph-mon[33019]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T23:41:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:05 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:05.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-04-24T23:41:05.933 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:41:06.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:06 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:41:06.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:06 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:41:06.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:06 smithi107 ceph-mon[29118]: osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] boot 2024-04-24T23:41:06.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:06 smithi107 ceph-mon[29118]: osdmap e67: 10 total, 10 up, 10 in 2024-04-24T23:41:06.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:06 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:06 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:41:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:06 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:41:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:06 smithi170 ceph-mon[32714]: osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] boot 2024-04-24T23:41:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:06 smithi170 ceph-mon[32714]: osdmap e67: 10 total, 10 up, 10 in 2024-04-24T23:41:06.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:06 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:06 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:41:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:06 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:41:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:06 smithi195 ceph-mon[33019]: osd.9 [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] boot 2024-04-24T23:41:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:06 smithi195 ceph-mon[33019]: osdmap e67: 10 total, 10 up, 10 in 2024-04-24T23:41:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:06 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T23:41:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:07 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T23:41:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:07 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 -- 172.21.15.170:0/198064803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010103490 msgr2=0x7f30101038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 --2- 172.21.15.170:0/198064803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010103490 0x7f30101038f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f30000099c0 tx=0x7f3000009cd0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 -- 172.21.15.170:0/198064803 shutdown_connections 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 --2- 172.21.15.170:0/198064803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 0x7f301010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 --2- 172.21.15.170:0/198064803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010102290 0x7f30101026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 --2- 172.21.15.170:0/198064803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010103490 0x7f30101038f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 -- 172.21.15.170:0/198064803 >> 172.21.15.170:0/198064803 conn(0x7f30100fd830 msgr2=0x7f30100ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 -- 172.21.15.170:0/198064803 shutdown_connections 2024-04-24T23:41:07.594 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.593+0000 7f301484f700 1 -- 172.21.15.170:0/198064803 wait complete. 2024-04-24T23:41:07.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.594+0000 7f301484f700 1 Processor -- start 2024-04-24T23:41:07.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.594+0000 7f301484f700 1 -- start start 2024-04-24T23:41:07.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.594+0000 7f301484f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:07.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.594+0000 7f301484f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010103490 0x7f3010198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:07.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 0x7f301019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f301010de70 con 0x7f3010103490 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f301010dcf0 con 0x7f3010102290 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f301010dff0 con 0x7f3010103e30 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:37794/0 (socket says 172.21.15.170:37794) 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 -- 172.21.15.170:0/3763849181 learned_addr learned my addr 172.21.15.170:0/3763849181 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300dd9b700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010103490 0x7f3010198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300ed9d700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 0x7f301019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 -- 172.21.15.170:0/3763849181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 msgr2=0x7f301019faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 0x7f301019faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 -- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010103490 msgr2=0x7f3010198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010103490 0x7f3010198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 -- 172.21.15.170:0/3763849181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30101a0140 con 0x7f3010102290 2024-04-24T23:41:07.597 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f300e59c700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f300400e970 tx=0x7f300400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300400cbd0 con 0x7f3010102290 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3004004500 con 0x7f3010102290 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30101a0430 con 0x7f3010102290 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004003e30 con 0x7f3010102290 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.595+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30101a0970 con 0x7f3010102290 2024-04-24T23:41:07.598 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.596+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3010068f20 con 0x7f3010102290 2024-04-24T23:41:07.599 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.597+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3004004000 con 0x7f3010102290 2024-04-24T23:41:07.599 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.598+0000 7f2fff7fe700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 0x7f2ff80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:07.599 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.598+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f30040cecd0 con 0x7f3010102290 2024-04-24T23:41:07.599 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.598+0000 7f300dd9b700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 0x7f2ff80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:07.599 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.599+0000 7f300dd9b700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 0x7f2ff80a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:07.601 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.601+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f300401f340 con 0x7f3010102290 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:07.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:07 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:07.805 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:07.805+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3010062670 con 0x7f2ff80a2c20 2024-04-24T23:41:08.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:08 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi107 2024-04-24T23:41:08.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:08 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi107 to 4272M 2024-04-24T23:41:08.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:08.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:08.515 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:08 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:08 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi107 2024-04-24T23:41:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:08 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi107 to 4272M 2024-04-24T23:41:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:08 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:08.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:08 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi107 2024-04-24T23:41:08.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:08 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi107 to 4272M 2024-04-24T23:41:08.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:08.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:08.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:08 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:09.390 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:09 smithi170 ceph-mon[32714]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:09.390 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:09 smithi170 ceph-mon[32714]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:09 smithi195 ceph-mon[33019]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:09 smithi195 ceph-mon[33019]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:09 smithi107 ceph-mon[29118]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:09 smithi107 ceph-mon[29118]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/3055448968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]': finished 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: osdmap e69: 11 total, 10 up, 11 in 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:11 smithi195 ceph-mon[33019]: from='client.? 172.21.15.170:0/186880193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/3055448968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]': finished 2024-04-24T23:41:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: osdmap e69: 11 total, 10 up, 11 in 2024-04-24T23:41:11.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:11.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:11 smithi170 ceph-mon[32714]: from='client.? 172.21.15.170:0/186880193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/3055448968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]: dispatch 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6b39d-cb02-4439-9562-fb2ee00ed40d"}]': finished 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: osdmap e69: 11 total, 10 up, 11 in 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:11.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:11 smithi107 ceph-mon[29118]: from='client.? 172.21.15.170:0/186880193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:13 smithi195 ceph-mon[33019]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:13 smithi170 ceph-mon[32714]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:13.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:13 smithi107 ceph-mon[29118]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:15 smithi195 ceph-mon[33019]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:15 smithi170 ceph-mon[32714]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:15.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:15 smithi107 ceph-mon[29118]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:17.546 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:17 smithi170 ceph-mon[32714]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:17 smithi195 ceph-mon[33019]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:17.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:17 smithi107 ceph-mon[29118]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:19 smithi195 ceph-mon[33019]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T23:41:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:19 smithi170 ceph-mon[32714]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T23:41:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:19.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:19 smithi107 ceph-mon[29118]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:19.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T23:41:19.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:20.511 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:20 smithi170 ceph-mon[32714]: Deploying daemon osd.10 on smithi170 2024-04-24T23:41:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:20 smithi195 ceph-mon[33019]: Deploying daemon osd.10 on smithi170 2024-04-24T23:41:20.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:20 smithi107 ceph-mon[29118]: Deploying daemon osd.10 on smithi170 2024-04-24T23:41:21.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:21 smithi170 ceph-mon[32714]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:21.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:21 smithi107 ceph-mon[29118]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:21 smithi195 ceph-mon[33019]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:23 smithi195 ceph-mon[33019]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:23 smithi170 ceph-mon[32714]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:23.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:23 smithi107 ceph-mon[29118]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:25.438 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:25 smithi170 ceph-mon[32714]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:25.438 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.438 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.438 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:25 smithi195 ceph-mon[33019]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:25 smithi107 ceph-mon[29118]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:27 smithi195 ceph-mon[33019]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:27.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:27 smithi170 ceph-mon[32714]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:27.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:27 smithi107 ceph-mon[29118]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:28.422 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 10 on host 'smithi170' 2024-04-24T23:41:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.422+0000 7f2fff7fe700 1 -- 172.21.15.170:0/3763849181 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3010062670 con 0x7f2ff80a2c20 2024-04-24T23:41:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 msgr2=0x7f2ff80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 0x7f2ff80a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005fb0 comp rx=0 tx=0).stop 2024-04-24T23:41:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 msgr2=0x7f30101984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f300400e970 tx=0x7f300400ec80 comp rx=0 tx=0).stop 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 shutdown_connections 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f2ff80a2c20 0x7f2ff80a50e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3010103e30 0x7f301019faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3010102290 0x7f30101984e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 --2- 172.21.15.170:0/3763849181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3010103490 0x7f3010198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 >> 172.21.15.170:0/3763849181 conn(0x7f30100fd830 msgr2=0x7f30101067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 shutdown_connections 2024-04-24T23:41:28.438 INFO:teuthology.orchestra.run.smithi170.stderr:2024-04-24T23:41:28.424+0000 7f301484f700 1 -- 172.21.15.170:0/3763849181 wait complete. 2024-04-24T23:41:28.730 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.731 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:28.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:28.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:28.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.057 DEBUG:teuthology.orchestra.run.smithi170:osd.10> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.10.service 2024-04-24T23:41:29.061 INFO:tasks.cephadm:Deploying osd.11 on smithi195 with /dev/vg_nvme/lv_1... 2024-04-24T23:41:29.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:29.317 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:41:29.618 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:29 smithi195 ceph-mon[33019]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T23:41:29.618 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:29 smithi195 ceph-mon[33019]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:29.618 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:29 smithi195 ceph-mon[33019]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T23:41:29.618 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:29 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:29.618 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:29 smithi195 ceph-mon[33019]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:29.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-mon[32714]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T23:41:29.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-mon[32714]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:29.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-mon[32714]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T23:41:29.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:29.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-mon[32714]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:29.836 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:41:29.605+0000 7f42b1d1c700 -1 osd.10 0 waiting for initial osdmap 2024-04-24T23:41:29.836 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:41:29 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:41:29.617+0000 7f42adb0f700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:41:29.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:29 smithi107 ceph-mon[29118]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T23:41:29.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:29 smithi107 ceph-mon[29118]: from='osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:29.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:29 smithi107 ceph-mon[29118]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T23:41:29.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:29 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:29.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:29 smithi107 ceph-mon[29118]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] boot 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: osdmap e72: 11 total, 11 up, 11 in 2024-04-24T23:41:30.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:30 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.871 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:41:30.871 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] boot 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: osdmap e72: 11 total, 11 up, 11 in 2024-04-24T23:41:30.872 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:30 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: osd.10 [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] boot 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: osdmap e72: 11 total, 11 up, 11 in 2024-04-24T23:41:30.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:30 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00869646 s, 1.2 GB/s 2024-04-24T23:41:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-04-24T23:41:31.797 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:31 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:41:31.798 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:31 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:41:31.798 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:31 smithi170 ceph-mon[32714]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T23:41:31.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:31 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:41:31.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:31 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:41:31.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:31 smithi107 ceph-mon[29118]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T23:41:31.917 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:31 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:41:31.917 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:31 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:41:31.918 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:31 smithi195 ceph-mon[33019]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T23:41:32.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_1 2024-04-24T23:41:32.610 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi170 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi170 to 4272M 2024-04-24T23:41:33.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:33.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:33.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:33 smithi195 ceph-mon[33019]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T23:41:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi170 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi170 to 4272M 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:33 smithi170 ceph-mon[32714]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi170 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi170 to 4272M 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:33 smithi107 ceph-mon[29118]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T23:41:34.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 -- 172.21.15.195:0/3041655170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4101eb0 msgr2=0x7f0bd4102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:34.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/3041655170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4101eb0 0x7f0bd4102330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009990 tx=0x7f0bcc009ca0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 -- 172.21.15.195:0/3041655170 shutdown_connections 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/3041655170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4101eb0 0x7f0bd4102330 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/3041655170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd4100b90 0x7f0bd4100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/3041655170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd41014f0 0x7f0bd4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 -- 172.21.15.195:0/3041655170 >> 172.21.15.195:0/3041655170 conn(0x7f0bd40fc730 msgr2=0x7f0bd40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 -- 172.21.15.195:0/3041655170 shutdown_connections 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.319+0000 7f0bdbe35700 1 -- 172.21.15.195:0/3041655170 wait complete. 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 Processor -- start 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 -- start start 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd41014f0 0x7f0bd419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd4101eb0 0x7f0bd41a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4110070 con 0x7f0bd4101eb0 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd410fef0 con 0x7f0bd41014f0 2024-04-24T23:41:34.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bdbe35700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd41101f0 con 0x7f0bd4100b90 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:54966/0 (socket says 172.21.15.195:54966) 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 -- 172.21.15.195:0/478519314 learned_addr learned my addr 172.21.15.195:0/478519314 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd93d0700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd41014f0 0x7f0bd419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bda3d2700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd4101eb0 0x7f0bd41a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 -- 172.21.15.195:0/478519314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd41014f0 msgr2=0x7f0bd419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd41014f0 0x7f0bd419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 -- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd4101eb0 msgr2=0x7f0bd41a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd4101eb0 0x7f0bd41a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.320+0000 7f0bd9bd1700 1 -- 172.21.15.195:0/478519314 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd41a2310 con 0x7f0bd4100b90 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bd9bd1700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4007f40 tx=0x7f0bc400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4004bb0 con 0x7f0bd4100b90 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd41a2600 con 0x7f0bd4100b90 2024-04-24T23:41:34.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc4004d50 con 0x7f0bd4100b90 2024-04-24T23:41:34.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc40133f0 con 0x7f0bd4100b90 2024-04-24T23:41:34.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.321+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd41a2b40 con 0x7f0bd4100b90 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.322+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd4068f20 con 0x7f0bd4100b90 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.323+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0bc401a020 con 0x7f0bd4100b90 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.324+0000 7f0bd2ffd700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 0x7f0bbc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.324+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f0bc40ce8a0 con 0x7f0bd4100b90 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.324+0000 7f0bd93d0700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 0x7f0bbc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:34.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.325+0000 7f0bd93d0700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 0x7f0bbc0a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8007be0 tx=0x7f0bc80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:34.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.327+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0bc4010070 con 0x7f0bd4100b90 2024-04-24T23:41:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:34.532+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0bd4062670 con 0x7f0bbc0a2c20 2024-04-24T23:41:34.533 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:34 smithi195 ceph-mon[33019]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T23:41:34.533 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:34 smithi195 ceph-mon[33019]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:34 smithi170 ceph-mon[32714]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T23:41:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:34 smithi170 ceph-mon[32714]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:34.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:34 smithi107 ceph-mon[29118]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T23:41:34.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:34 smithi107 ceph-mon[29118]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:35.392 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:35 smithi195 ceph-mon[33019]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T23:41:35.392 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:35 smithi195 ceph-mon[33019]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:35.392 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:35.392 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:35.392 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:35 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:35 smithi170 ceph-mon[32714]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T23:41:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:35 smithi170 ceph-mon[32714]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:35 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:35 smithi107 ceph-mon[29118]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T23:41:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:35 smithi107 ceph-mon[29118]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:41:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T23:41:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T23:41:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:35 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:36.499 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:36 smithi195 ceph-mon[33019]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-24T23:41:36.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:36 smithi170 ceph-mon[32714]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-24T23:41:36.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:36 smithi107 ceph-mon[29118]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-24T23:41:37.682 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:37 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/251507652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.682 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:37 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.683 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:37 smithi195 ceph-mon[33019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]': finished 2024-04-24T23:41:37.683 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:37 smithi195 ceph-mon[33019]: osdmap e77: 12 total, 11 up, 12 in 2024-04-24T23:41:37.683 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:37.758 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:37 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/251507652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.758 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:37 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.759 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:37 smithi170 ceph-mon[32714]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]': finished 2024-04-24T23:41:37.759 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:37 smithi170 ceph-mon[32714]: osdmap e77: 12 total, 11 up, 12 in 2024-04-24T23:41:37.759 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:37 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:37 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/251507652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:37 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]: dispatch 2024-04-24T23:41:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:37 smithi107 ceph-mon[29118]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc99798e-5b23-45a7-9bf3-40c43d5d092e"}]': finished 2024-04-24T23:41:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:37 smithi107 ceph-mon[29118]: osdmap e77: 12 total, 11 up, 12 in 2024-04-24T23:41:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:38.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:38 smithi170 ceph-mon[32714]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-24T23:41:38.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:38 smithi170 ceph-mon[32714]: from='client.? 172.21.15.195:0/1952245220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:38 smithi195 ceph-mon[33019]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-24T23:41:38.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:38 smithi195 ceph-mon[33019]: from='client.? 172.21.15.195:0/1952245220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:38 smithi107 ceph-mon[29118]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-24T23:41:38.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:38 smithi107 ceph-mon[29118]: from='client.? 172.21.15.195:0/1952245220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T23:41:41.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:41 smithi195 ceph-mon[33019]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:41:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:41 smithi170 ceph-mon[32714]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:41:41.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:41 smithi107 ceph-mon[29118]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T23:41:43.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:43 smithi195 ceph-mon[33019]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:41:43.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:43 smithi170 ceph-mon[32714]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:41:43.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:43 smithi107 ceph-mon[29118]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T23:41:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:45 smithi195 ceph-mon[33019]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T23:41:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T23:41:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:45 smithi170 ceph-mon[32714]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T23:41:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T23:41:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:45 smithi107 ceph-mon[29118]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T23:41:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T23:41:45.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:46 smithi195 ceph-mon[33019]: Deploying daemon osd.11 on smithi195 2024-04-24T23:41:46.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:46 smithi170 ceph-mon[32714]: Deploying daemon osd.11 on smithi195 2024-04-24T23:41:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:46 smithi107 ceph-mon[29118]: Deploying daemon osd.11 on smithi195 2024-04-24T23:41:47.207 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:47 smithi195 ceph-mon[33019]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:47 smithi170 ceph-mon[32714]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:47.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:47 smithi107 ceph-mon[29118]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:49.116 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:49 smithi195 ceph-mon[33019]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:49.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:49 smithi170 ceph-mon[32714]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:49 smithi107 ceph-mon[29118]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:51 smithi195 ceph-mon[33019]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:51 smithi170 ceph-mon[32714]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:51.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:51 smithi107 ceph-mon[29118]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:51.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:51.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:53.289 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:53 smithi195 ceph-mon[33019]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:53 smithi170 ceph-mon[32714]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:53.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:53 smithi107 ceph-mon[29118]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:54.548 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.548+0000 7f0bd2ffd700 1 -- 172.21.15.195:0/478519314 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0bd4062670 con 0x7f0bbc0a2c20 2024-04-24T23:41:54.549 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 11 on host 'smithi195' 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 msgr2=0x7f0bbc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 0x7f0bbc0a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8007be0 tx=0x7f0bc80073d0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 msgr2=0x7f0bd419a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4007f40 tx=0x7f0bc400d3b0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 shutdown_connections 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0bbc0a2c20 0x7f0bbc0a50e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd4101eb0 0x7f0bd41a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0bd4100b90 0x7f0bd419a6b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 --2- 172.21.15.195:0/478519314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bd41014f0 0x7f0bd419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 >> 172.21.15.195:0/478519314 conn(0x7f0bd40fc730 msgr2=0x7f0bd40fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:54.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 shutdown_connections 2024-04-24T23:41:54.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:41:54.551+0000 7f0bdbe35700 1 -- 172.21.15.195:0/478519314 wait complete. 2024-04-24T23:41:55.142 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:55 smithi195 ceph-mon[33019]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:55.143 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:55 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.143 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:55 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.143 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:55 smithi195 ceph-mon[33019]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.143 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:55 smithi195 ceph-mon[33019]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.169 DEBUG:teuthology.orchestra.run.smithi195:osd.11> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.11.service 2024-04-24T23:41:55.171 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-24T23:41:55.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-24T23:41:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:55 smithi170 ceph-mon[32714]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:55 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:55 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:55 smithi170 ceph-mon[32714]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:55 smithi170 ceph-mon[32714]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:55 smithi107 ceph-mon[29118]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:55.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:55 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:55 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:55 smithi107 ceph-mon[29118]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:55 smithi107 ceph-mon[29118]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T23:41:55.418 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.607 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-mon[33019]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:56.608 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:41:56.535+0000 7f815c5fd700 -1 osd.11 0 waiting for initial osdmap 2024-04-24T23:41:56.608 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:41:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:41:56.550+0000 7f81583f0700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:41:56.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T23:41:56.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T23:41:56.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:56.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:56 smithi170 ceph-mon[32714]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:56.850 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:56.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:56 smithi107 ceph-mon[29118]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 -- 172.21.15.107:0/2479555986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1033b0 msgr2=0x7f24ec103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 --2- 172.21.15.107:0/2479555986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1033b0 0x7f24ec103810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f24e00099c0 tx=0x7f24e0009cd0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 -- 172.21.15.107:0/2479555986 shutdown_connections 2024-04-24T23:41:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 --2- 172.21.15.107:0/2479555986 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec103d50 0x7f24ec10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 --2- 172.21.15.107:0/2479555986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec1021b0 0x7f24ec1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 --2- 172.21.15.107:0/2479555986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1033b0 0x7f24ec103810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 -- 172.21.15.107:0/2479555986 >> 172.21.15.107:0/2479555986 conn(0x7f24ec0fd770 msgr2=0x7f24ec0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 -- 172.21.15.107:0/2479555986 shutdown_connections 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 -- 172.21.15.107:0/2479555986 wait complete. 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.251+0000 7f24f099d700 1 Processor -- start 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 -- start start 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec1033b0 0x7f24ec198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 0x7f24ec19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10de90 con 0x7f24ec1021b0 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10dd10 con 0x7f24ec103d50 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10e010 con 0x7f24ec1033b0 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50634/0 (socket says 172.21.15.107:50634) 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 -- 172.21.15.107:0/1157404339 learned_addr learned my addr 172.21.15.107:0/1157404339 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eaffd700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec1033b0 0x7f24ec198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24ebfff700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 0x7f24ec19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 -- 172.21.15.107:0/1157404339 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec1033b0 msgr2=0x7f24ec198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec1033b0 0x7f24ec198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 -- 172.21.15.107:0/1157404339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 msgr2=0x7f24ec19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 0x7f24ec19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 -- 172.21.15.107:0/1157404339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ec1a0160 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24ebfff700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 0x7f24ec19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24eb7fe700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00b530 tx=0x7f24dc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc015020 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ec1a0450 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24dc011d10 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.252+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00f2e0 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.253+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ec1a0a10 con 0x7f24ec1021b0 2024-04-24T23:41:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.253+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24ec068f20 con 0x7f24ec1021b0 2024-04-24T23:41:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.255+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f24dc018020 con 0x7f24ec1021b0 2024-04-24T23:41:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.256+0000 7f24e8ff9700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 0x7f24d40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:41:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.256+0000 7f24eaffd700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 0x7f24d40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:41:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.256+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f24dc0ce640 con 0x7f24ec1021b0 2024-04-24T23:41:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.256+0000 7f24eaffd700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 0x7f24d40a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:41:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.259+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f24dc01f020 con 0x7f24ec1021b0 2024-04-24T23:41:57.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.459+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f24ec02db30 con 0x7f24ec1021b0 2024-04-24T23:41:57.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.459+0000 7f24e8ff9700 1 -- 172.21.15.107:0/1157404339 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f24dc09bb60 con 0x7f24ec1021b0 2024-04-24T23:41:57.459 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:41:57.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 msgr2=0x7f24d40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:57.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 0x7f24d40a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e0005fb0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 msgr2=0x7f24ec198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:41:57.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00b530 tx=0x7f24dc00b840 comp rx=0 tx=0).stop 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 shutdown_connections 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24d40a2c20 0x7f24d40a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ec103d50 0x7f24ec19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24ec1021b0 0x7f24ec198500 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 --2- 172.21.15.107:0/1157404339 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24ec1033b0 0x7f24ec198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.462+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 >> 172.21.15.107:0/1157404339 conn(0x7f24ec0fd770 msgr2=0x7f24ec106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.463+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 shutdown_connections 2024-04-24T23:41:57.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:41:57.463+0000 7f24f099d700 1 -- 172.21.15.107:0/1157404339 wait complete. 2024-04-24T23:41:57.638 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:57 smithi195 ceph-mon[33019]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:41:57.638 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:57 smithi195 ceph-mon[33019]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T23:41:57.638 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:57 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.638 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:57 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.639 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:57 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1157404339' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:41:57.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:57 smithi107 ceph-mon[29118]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:41:57.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:57 smithi107 ceph-mon[29118]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T23:41:57.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:57 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:57 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1157404339' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:41:57.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:57 smithi170 ceph-mon[32714]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-04-24T23:41:57.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:57 smithi170 ceph-mon[32714]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T23:41:57.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:57 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:57 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:57.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:57 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1157404339' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:41:58.219 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714002090,"num_in_osds":12,"osd_in_since":1714002097,"num_remapped_pgs":0} 2024-04-24T23:41:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: purged_snaps scrub starts 2024-04-24T23:41:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: purged_snaps scrub ok 2024-04-24T23:41:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] boot 2024-04-24T23:41:58.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: osdmap e80: 12 total, 12 up, 12 in 2024-04-24T23:41:58.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:58.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:58 smithi195 ceph-mon[33019]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:58.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: purged_snaps scrub starts 2024-04-24T23:41:58.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: purged_snaps scrub ok 2024-04-24T23:41:58.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] boot 2024-04-24T23:41:58.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: osdmap e80: 12 total, 12 up, 12 in 2024-04-24T23:41:58.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:58.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:58 smithi170 ceph-mon[32714]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:58.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: purged_snaps scrub starts 2024-04-24T23:41:58.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: purged_snaps scrub ok 2024-04-24T23:41:58.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: osd.11 [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] boot 2024-04-24T23:41:58.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: osdmap e80: 12 total, 12 up, 12 in 2024-04-24T23:41:58.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T23:41:58.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:58 smithi107 ceph-mon[29118]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-24T23:41:59.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-24T23:41:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:59.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:59.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:41:59 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:41:59 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:41:59.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:41:59 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: Detected new or changed devices on smithi195 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: Adjusting osd_memory_target on smithi195 to 4272M 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:00 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:00.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: Detected new or changed devices on smithi195 2024-04-24T23:42:00.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: Adjusting osd_memory_target on smithi195 to 4272M 2024-04-24T23:42:00.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:00.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:00.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:00.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:00 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:00.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: Detected new or changed devices on smithi195 2024-04-24T23:42:00.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: Adjusting osd_memory_target on smithi195 to 4272M 2024-04-24T23:42:00.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:00.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:00.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:00.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:00 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/1807524740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 msgr2=0x7fbd381015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/1807524740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd381015b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000b0d0 tx=0x7fbd3000b3e0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/1807524740 shutdown_connections 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/1807524740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102d30 0x7fbd381095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/1807524740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd381015b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/1807524740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102390 0x7fbd381027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/1807524740 >> 172.21.15.107:0/1807524740 conn(0x7fbd380fc730 msgr2=0x7fbd380feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/1807524740 shutdown_connections 2024-04-24T23:42:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.136+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/1807524740 wait complete. 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 Processor -- start 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 -- start start 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 0x7fbd3819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 0x7fbd3819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3810cd70 con 0x7fbd38101190 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3810cbf0 con 0x7fbd38102390 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3ffc4700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3810cef0 con 0x7fbd38102d30 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50646/0 (socket says 172.21.15.107:50646) 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 -- 172.21.15.107:0/474685500 learned_addr learned my addr 172.21.15.107:0/474685500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3e561700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 0x7fbd3819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3d55f700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 0x7fbd3819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 -- 172.21.15.107:0/474685500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 msgr2=0x7fbd3819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 0x7fbd3819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 -- 172.21.15.107:0/474685500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 msgr2=0x7fbd3819d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 0x7fbd3819d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3dd60700 1 -- 172.21.15.107:0/474685500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381a0150 con 0x7fbd38101190 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3d55f700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 0x7fbd3819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.137+0000 7fbd3e561700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 0x7fbd3819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd3dd60700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007d10 tx=0x7fbd30007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3001f070 con 0x7fbd38101190 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd38197010 con 0x7fbd38101190 2024-04-24T23:42:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd30003e10 con 0x7fbd38101190 2024-04-24T23:42:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3001b750 con 0x7fbd38101190 2024-04-24T23:42:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.138+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd38197540 con 0x7fbd38101190 2024-04-24T23:42:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.139+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd38068f20 con 0x7fbd38101190 2024-04-24T23:42:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.140+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbd30022070 con 0x7fbd38101190 2024-04-24T23:42:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.140+0000 7fbd36ffd700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 0x7fbd200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.140+0000 7fbd3d55f700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 0x7fbd200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.140+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbd3000f020 con 0x7fbd38101190 2024-04-24T23:42:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.141+0000 7fbd3d55f700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 0x7fbd200a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c005fd0 tx=0x7fbd2c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.145+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd30029020 con 0x7fbd38101190 2024-04-24T23:42:01.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.342+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fbd3802db30 con 0x7fbd38101190 2024-04-24T23:42:01.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.342+0000 7fbd36ffd700 1 -- 172.21.15.107:0/474685500 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fbd3009d2f0 con 0x7fbd38101190 2024-04-24T23:42:01.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:01.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 msgr2=0x7fbd200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:01.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 0x7fbd200a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c005fd0 tx=0x7fbd2c005ee0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 msgr2=0x7fbd3806aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007d10 tx=0x7fbd30007df0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 shutdown_connections 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbd200a2c20 0x7fbd200a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbd38102d30 0x7fbd3819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38101190 0x7fbd3806aa70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 --2- 172.21.15.107:0/474685500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd38102390 0x7fbd3819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.344+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 >> 172.21.15.107:0/474685500 conn(0x7fbd380fc730 msgr2=0x7fbd38105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.345+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 shutdown_connections 2024-04-24T23:42:01.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:01.345+0000 7fbd3ffc4700 1 -- 172.21.15.107:0/474685500 wait complete. 2024-04-24T23:42:01.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:01 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/474685500' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:42:01.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:01 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/474685500' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:42:01.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/474685500' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T23:42:01.900 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714002117,"num_in_osds":12,"osd_in_since":1714002097,"num_remapped_pgs":0} 2024-04-24T23:42:01.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T23:42:02.131 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:42:02.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:02 smithi170 ceph-mon[32714]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:02.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:02 smithi107 ceph-mon[29118]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:03.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:02 smithi195 ceph-mon[33019]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 -- 172.21.15.107:0/19886607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b01075a0 msgr2=0x7f15b010f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 --2- 172.21.15.107:0/19886607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b01075a0 0x7f15b010f150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15a800b0a0 tx=0x7f15a800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 -- 172.21.15.107:0/19886607 shutdown_connections 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 --2- 172.21.15.107:0/19886607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b01075a0 0x7f15b010f150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 --2- 172.21.15.107:0/19886607 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006ac00 0x7f15b006afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 --2- 172.21.15.107:0/19886607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b006b5b0 0x7f15b0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 -- 172.21.15.107:0/19886607 >> 172.21.15.107:0/19886607 conn(0x7f15b0077180 msgr2=0x7f15b0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 -- 172.21.15.107:0/19886607 shutdown_connections 2024-04-24T23:42:03.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.757+0000 7f15b8653700 1 -- 172.21.15.107:0/19886607 wait complete. 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 Processor -- start 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 -- start start 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006b5b0 0x7f15b019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b01075a0 0x7f15b0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01125c0 con 0x7f15b006ac00 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0112440 con 0x7f15b01075a0 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b8653700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0112740 con 0x7f15b006b5b0 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50656/0 (socket says 172.21.15.107:50656) 2024-04-24T23:42:03.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 -- 172.21.15.107:0/1097635794 learned_addr learned my addr 172.21.15.107:0/1097635794 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b6bf0700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b01075a0 0x7f15b0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b5bee700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006b5b0 0x7f15b019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 -- 172.21.15.107:0/1097635794 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006b5b0 msgr2=0x7f15b019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006b5b0 0x7f15b019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 -- 172.21.15.107:0/1097635794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b01075a0 msgr2=0x7f15b0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b01075a0 0x7f15b0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.758+0000 7f15b63ef700 1 -- 172.21.15.107:0/1097635794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b0199020 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15b63ef700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00c9b0 tx=0x7f15ac00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac007b00 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ac007ca0 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b0199310 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac018730 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.759+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b01a55f0 con 0x7f15b006ac00 2024-04-24T23:42:03.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.760+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b01039a0 con 0x7f15b006ac00 2024-04-24T23:42:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.762+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f15ac0188d0 con 0x7f15b006ac00 2024-04-24T23:42:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.762+0000 7f15a77fe700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 0x7f159c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.762+0000 7f15b5bee700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 0x7f159c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:03.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.762+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15ac0cee00 con 0x7f15b006ac00 2024-04-24T23:42:03.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.763+0000 7f15b5bee700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 0x7f159c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f15a0006fd0 tx=0x7f15a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:03.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.766+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15ac014030 con 0x7f15b006ac00 2024-04-24T23:42:03.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.963+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15b01a57c0 con 0x7f15b006ac00 2024-04-24T23:42:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.964+0000 7f15a77fe700 1 -- 172.21.15.107:0/1097635794 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7f15ac09c290 con 0x7f15b006ac00 2024-04-24T23:42:03.964 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:03.965 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:41:58.539760+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 msgr2=0x7f159c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 0x7f159c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f15a0006fd0 tx=0x7f15a0008040 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 msgr2=0x7f15b019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00c9b0 tx=0x7f15ac00ccc0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 shutdown_connections 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f159c0a2c20 0x7f159c0a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b01075a0 0x7f15b0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15b006ac00 0x7f15b019e670 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 --2- 172.21.15.107:0/1097635794 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15b006b5b0 0x7f15b019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.966+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 >> 172.21.15.107:0/1097635794 conn(0x7f15b0077180 msgr2=0x7f15b00ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.967+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 shutdown_connections 2024-04-24T23:42:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:03.967+0000 7f15b8653700 1 -- 172.21.15.107:0/1097635794 wait complete. 2024-04-24T23:42:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1097635794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:04.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1097635794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:04.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1097635794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:04.570 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:42:04.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-24T23:42:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:42:05.240 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:05 smithi107 ceph-mon[29118]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:05 smithi195 ceph-mon[33019]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:05.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:05 smithi170 ceph-mon[32714]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 -- 172.21.15.107:0/4099486460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 msgr2=0x7f23ac10f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 --2- 172.21.15.107:0/4099486460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac10f150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f23a400b0a0 tx=0x7f23a400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 -- 172.21.15.107:0/4099486460 shutdown_connections 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 --2- 172.21.15.107:0/4099486460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac10f150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 --2- 172.21.15.107:0/4099486460 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 0x7f23ac06afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 --2- 172.21.15.107:0/4099486460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 0x7f23ac106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 -- 172.21.15.107:0/4099486460 >> 172.21.15.107:0/4099486460 conn(0x7f23ac077180 msgr2=0x7f23ac077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 -- 172.21.15.107:0/4099486460 shutdown_connections 2024-04-24T23:42:06.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 -- 172.21.15.107:0/4099486460 wait complete. 2024-04-24T23:42:06.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.552+0000 7f23b11cd700 1 Processor -- start 2024-04-24T23:42:06.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 -- start start 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 0x7f23ac1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 0x7f23ac1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac112560 con 0x7f23ac1075a0 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1123e0 con 0x7f23ac06b5b0 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23b11cd700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1126e0 con 0x7f23ac06ac00 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54944/0 (socket says 172.21.15.107:54944) 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 -- 172.21.15.107:0/1034041480 learned_addr learned my addr 172.21.15.107:0/1034041480 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23aad9d700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 0x7f23ac1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23a259c700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 0x7f23ac1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 -- 172.21.15.107:0/1034041480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 msgr2=0x7f23ac1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 0x7f23ac1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 -- 172.21.15.107:0/1034041480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 msgr2=0x7f23ac1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 0x7f23ac1a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 -- 172.21.15.107:0/1034041480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ac19b080 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.553+0000 7f23ab59e700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f23a40062a0 tx=0x7f23a4004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.554+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4021070 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.554+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ac19b310 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.554+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ac1a7630 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.554+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a4007c00 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.554+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a401d450 con 0x7f23ac1075a0 2024-04-24T23:42:06.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.555+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ac068f20 con 0x7f23ac1075a0 2024-04-24T23:42:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.556+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f23a4004000 con 0x7f23ac1075a0 2024-04-24T23:42:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.556+0000 7f23a8d99700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 0x7f23980a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.556+0000 7f23aad9d700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 0x7f23980a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.556+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f23a400f020 con 0x7f23ac1075a0 2024-04-24T23:42:06.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.557+0000 7f23aad9d700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 0x7f23980a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac19ba60 tx=0x7f239c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.560+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23a402a030 con 0x7f23ac1075a0 2024-04-24T23:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.752+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f23ac1a78c0 con 0x7f23ac1075a0 2024-04-24T23:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.753+0000 7f23a8d99700 1 -- 172.21.15.107:0/1034041480 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f23a409d5c0 con 0x7f23ac1075a0 2024-04-24T23:42:06.753 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 msgr2=0x7f23980a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 0x7f23980a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac19ba60 tx=0x7f239c00b410 comp rx=0 tx=0).stop 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 msgr2=0x7f23ac19a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f23a40062a0 tx=0x7f23a4004d70 comp rx=0 tx=0).stop 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 shutdown_connections 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f23980a2ca0 0x7f23980a5160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23ac1075a0 0x7f23ac19a920 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f23ac06ac00 0x7f23ac1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 --2- 172.21.15.107:0/1034041480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23ac06b5b0 0x7f23ac1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:06.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 >> 172.21.15.107:0/1034041480 conn(0x7f23ac077180 msgr2=0x7f23ac10d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:06.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 shutdown_connections 2024-04-24T23:42:06.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:06.755+0000 7f23b11cd700 1 -- 172.21.15.107:0/1034041480 wait complete. 2024-04-24T23:42:07.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:07 smithi195 ceph-mon[33019]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:07.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1034041480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:07 smithi170 ceph-mon[32714]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1034041480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:07.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:07 smithi107 ceph-mon[29118]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:07.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1034041480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.a on smithi107 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.d on smithi107 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.g on smithi107 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.j on smithi107 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.b on smithi170 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.e on smithi170 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.h on smithi170 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.k on smithi170 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.c on smithi195 2024-04-24T23:42:07.547 INFO:tasks.cephadm:Adding mds.f on smithi195 2024-04-24T23:42:07.548 INFO:tasks.cephadm:Adding mds.i on smithi195 2024-04-24T23:42:07.548 INFO:tasks.cephadm:Adding mds.l on smithi195 2024-04-24T23:42:07.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l' 2024-04-24T23:42:07.795 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:42:09.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:09 smithi195 ceph-mon[33019]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:09.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:09 smithi170 ceph-mon[32714]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:09.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:09 smithi107 ceph-mon[29118]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 -- 172.21.15.195:0/3404279033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb01088d0 msgr2=0x7f6bb0108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 --2- 172.21.15.195:0/3404279033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb01088d0 0x7f6bb0108cf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009990 tx=0x7f6ba0009ca0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 -- 172.21.15.195:0/3404279033 shutdown_connections 2024-04-24T23:42:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 --2- 172.21.15.195:0/3404279033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb010a470 0x7f6bb0110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 --2- 172.21.15.195:0/3404279033 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb01088d0 0x7f6bb0108cf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 --2- 172.21.15.195:0/3404279033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb0109ad0 0x7f6bb0109f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 -- 172.21.15.195:0/3404279033 >> 172.21.15.195:0/3404279033 conn(0x7f6bb00778a0 msgr2=0x7f6bb0079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:09.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 -- 172.21.15.195:0/3404279033 shutdown_connections 2024-04-24T23:42:09.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.433+0000 7f6bb546e700 1 -- 172.21.15.195:0/3404279033 wait complete. 2024-04-24T23:42:09.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 Processor -- start 2024-04-24T23:42:09.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 -- start start 2024-04-24T23:42:09.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb01088d0 0x7f6bb019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:09.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 0x7f6bb019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb0114490 con 0x7f6bb0109ad0 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb0114310 con 0x7f6bb01088d0 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6bb546e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb0114610 con 0x7f6bb010a470 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6baf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6baf7fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:54110/0 (socket says 172.21.15.195:54110) 2024-04-24T23:42:09.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6baf7fe700 1 -- 172.21.15.195:0/2706049751 learned_addr learned my addr 172.21.15.195:0/2706049751 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.434+0000 7f6baeffd700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb01088d0 0x7f6bb019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6bae7fc700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 0x7f6bb019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 -- 172.21.15.195:0/2706049751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb01088d0 msgr2=0x7f6bb019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb01088d0 0x7f6bb019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 -- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 msgr2=0x7f6bb019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 0x7f6bb019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 -- 172.21.15.195:0/2706049751 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb01a6700 con 0x7f6bb010a470 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6bae7fc700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 0x7f6bb019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6baf7fe700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800add0 tx=0x7f6ba800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8017020 con 0x7f6bb010a470 2024-04-24T23:42:09.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba800ce20 con 0x7f6bb010a470 2024-04-24T23:42:09.440 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb01a69f0 con 0x7f6bb010a470 2024-04-24T23:42:09.440 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba80053e0 con 0x7f6bb010a470 2024-04-24T23:42:09.440 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.435+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb01a6ee0 con 0x7f6bb010a470 2024-04-24T23:42:09.440 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.436+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb0068f20 con 0x7f6bb010a470 2024-04-24T23:42:09.440 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.437+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6ba800c940 con 0x7f6bb010a470 2024-04-24T23:42:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.438+0000 7f6b97fff700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 0x7f6b980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.438+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6ba80cea30 con 0x7f6bb010a470 2024-04-24T23:42:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.438+0000 7f6baeffd700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 0x7f6b980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.439+0000 7f6baeffd700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 0x7f6b980a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0003890 tx=0x7f6ba0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:09.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.442+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6ba8021020 con 0x7f6bb010a470 2024-04-24T23:42:09.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.645+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l", "target": ["mon-mgr", ""]}) -- 0x7f6bb0062930 con 0x7f6b980a2c20 2024-04-24T23:42:09.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.653+0000 7f6b97fff700 1 -- 172.21.15.195:0/2706049751 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f6bb0062930 con 0x7f6b980a2c20 2024-04-24T23:42:09.653 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mds.all update... 2024-04-24T23:42:09.655 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 msgr2=0x7f6b980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 0x7f6b980a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0003890 tx=0x7f6ba0005fd0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 msgr2=0x7f6bb01a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800add0 tx=0x7f6ba800c330 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 shutdown_connections 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6b980a2c20 0x7f6b980a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6bb010a470 0x7f6bb01a6060 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bb01088d0 0x7f6bb019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 --2- 172.21.15.195:0/2706049751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bb0109ad0 0x7f6bb019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.655+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 >> 172.21.15.195:0/2706049751 conn(0x7f6bb00778a0 msgr2=0x7f6bb010cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.656+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 shutdown_connections 2024-04-24T23:42:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:2024-04-24T23:42:09.656+0000 7f6bb546e700 1 -- 172.21.15.195:0/2706049751 wait complete. 2024-04-24T23:42:10.331 DEBUG:teuthology.orchestra.run.smithi107:mds.a> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.a.service 2024-04-24T23:42:10.334 DEBUG:teuthology.orchestra.run.smithi107:mds.d> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.d.service 2024-04-24T23:42:10.378 DEBUG:teuthology.orchestra.run.smithi107:mds.g> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.g.service 2024-04-24T23:42:10.420 DEBUG:teuthology.orchestra.run.smithi107:mds.j> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.j.service 2024-04-24T23:42:10.463 DEBUG:teuthology.orchestra.run.smithi170:mds.b> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.b.service 2024-04-24T23:42:10.467 DEBUG:teuthology.orchestra.run.smithi170:mds.e> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.e.service 2024-04-24T23:42:10.510 DEBUG:teuthology.orchestra.run.smithi170:mds.h> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.h.service 2024-04-24T23:42:10.553 DEBUG:teuthology.orchestra.run.smithi170:mds.k> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.k.service 2024-04-24T23:42:10.595 DEBUG:teuthology.orchestra.run.smithi195:mds.c> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.c.service 2024-04-24T23:42:10.598 DEBUG:teuthology.orchestra.run.smithi195:mds.f> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.f.service 2024-04-24T23:42:10.641 DEBUG:teuthology.orchestra.run.smithi195:mds.i> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.i.service 2024-04-24T23:42:10.685 DEBUG:teuthology.orchestra.run.smithi195:mds.l> sudo journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.l.service 2024-04-24T23:42:10.728 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-24T23:42:10.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:42:10.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l;count:12 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:10.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:10.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:10.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:10.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: Deploying daemon mds.l on smithi195 2024-04-24T23:42:10.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:10 smithi170 ceph-mon[32714]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- 172.21.15.107:0/1489154344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0100690 msgr2=0x7fd8e0100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/1489154344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0100690 0x7fd8e0100ab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009900 tx=0x7fd8dc02a410 comp rx=0 tx=0).stop 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- 172.21.15.107:0/1489154344 shutdown_connections 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/1489154344 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0102240 0x7fd8e0108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/1489154344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0101880 0x7fd8e0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/1489154344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0100690 0x7fd8e0100ab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- 172.21.15.107:0/1489154344 >> 172.21.15.107:0/1489154344 conn(0x7fd8e00fbc20 msgr2=0x7fd8e00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:10.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- 172.21.15.107:0/1489154344 shutdown_connections 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- 172.21.15.107:0/1489154344 wait complete. 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 Processor -- start 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- start start 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0100690 0x7fd8e0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 0x7fd8e019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c360 con 0x7fd8e0101880 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c1e0 con 0x7fd8e0102240 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.845+0000 7fd8e86a7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c4e0 con 0x7fd8e0100690 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54964/0 (socket says 172.21.15.107:54964) 2024-04-24T23:42:10.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 -- 172.21.15.107:0/2449522987 learned_addr learned my addr 172.21.15.107:0/2449522987 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e6443640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0100690 0x7fd8e0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e6c44640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 0x7fd8e019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 -- 172.21.15.107:0/2449522987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0100690 msgr2=0x7fd8e0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0100690 0x7fd8e0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 -- 172.21.15.107:0/2449522987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 msgr2=0x7fd8e019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 0x7fd8e019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:10.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 -- 172.21.15.107:0/2449522987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e019f6e0 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e6c44640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 0x7fd8e019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e5c42640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000e9f0 tx=0x7fd8d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d000cd80 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8d0010910 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0018990 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e019f9d0 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.846+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e019ff20 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.847+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e01a0150 con 0x7fd8e0101880 2024-04-24T23:42:10.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.848+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd8d0010ac0 con 0x7fd8e0101880 2024-04-24T23:42:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.849+0000 7fd8d77fe640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 0x7fd8cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.849+0000 7fd8e6443640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 0x7fd8cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.849+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd8d00cef40 con 0x7fd8e0101880 2024-04-24T23:42:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.849+0000 7fd8e6443640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 0x7fd8cc0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc02aa00 tx=0x7fd8dc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:10.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:10.853+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8d009c340 con 0x7fd8e0101880 2024-04-24T23:42:11.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.027+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8e01a0440 con 0x7fd8e0101880 2024-04-24T23:42:11.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.028+0000 7fd8d77fe640 1 -- 172.21.15.107:0/2449522987 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7fd8d009bc40 con 0x7fd8e0101880 2024-04-24T23:42:11.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:11.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:41:58.539760+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 msgr2=0x7fd8cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 0x7fd8cc0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc02aa00 tx=0x7fd8dc0315b0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 msgr2=0x7fd8e0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000e9f0 tx=0x7fd8d000eeb0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 shutdown_connections 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd8cc0a2b90 0x7fd8cc0a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8e0102240 0x7fd8e019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8e0101880 0x7fd8e0198000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 --2- 172.21.15.107:0/2449522987 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8e0100690 0x7fd8e0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 >> 172.21.15.107:0/2449522987 conn(0x7fd8e00fbc20 msgr2=0x7fd8e00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 shutdown_connections 2024-04-24T23:42:11.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.030+0000 7fd8e86a7640 1 -- 172.21.15.107:0/2449522987 wait complete. 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l;count:12 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:11.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: Deploying daemon mds.l on smithi195 2024-04-24T23:42:11.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:10 smithi107 ceph-mon[29118]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:11.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:42:11.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l;count:12 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:11.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:11.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: Deploying daemon mds.l on smithi195 2024-04-24T23:42:11.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:10 smithi195 ceph-mon[33019]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 -- 172.21.15.107:0/3809057174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 msgr2=0x7f25700ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 --2- 172.21.15.107:0/3809057174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f25700ffe70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f256c009930 tx=0x7f256c02a420 comp rx=0 tx=0).stop 2024-04-24T23:42:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 -- 172.21.15.107:0/3809057174 shutdown_connections 2024-04-24T23:42:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 --2- 172.21.15.107:0/3809057174 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25701033e0 0x7f2570109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 --2- 172.21.15.107:0/3809057174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2570102a20 0x7f2570102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 --2- 172.21.15.107:0/3809057174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f25700ffe70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 -- 172.21.15.107:0/3809057174 >> 172.21.15.107:0/3809057174 conn(0x7f25700fb380 msgr2=0x7f25700fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 -- 172.21.15.107:0/3809057174 shutdown_connections 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.143+0000 7f25778f5640 1 -- 172.21.15.107:0/3809057174 wait complete. 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 Processor -- start 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 -- start start 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2570102a20 0x7f2570198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25701033e0 0x7f257019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f257010d450 con 0x7f25700ffa50 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f257010d2d0 con 0x7f2570102a20 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25778f5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f257010d5d0 con 0x7f25701033e0 2024-04-24T23:42:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54986/0 (socket says 172.21.15.107:54986) 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 -- 172.21.15.107:0/4049065191 learned_addr learned my addr 172.21.15.107:0/4049065191 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 -- 172.21.15.107:0/4049065191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25701033e0 msgr2=0x7f257019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25701033e0 0x7f257019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 -- 172.21.15.107:0/4049065191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2570102a20 msgr2=0x7f2570198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2570102a20 0x7f2570198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 -- 172.21.15.107:0/4049065191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019f710 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f2575691640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f256c031440 tx=0x7f256c032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c03f030 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.144+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256c003ce0 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.145+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c035070 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.145+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019f9a0 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.145+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257019fee0 con 0x7f25700ffa50 2024-04-24T23:42:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.145+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2570108bd0 con 0x7f25700ffa50 2024-04-24T23:42:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.146+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f256c042030 con 0x7f25700ffa50 2024-04-24T23:42:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.147+0000 7f25667fc640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 0x7f255c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.147+0000 7f2574e90640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 0x7f255c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.147+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f256c0ef290 con 0x7f25700ffa50 2024-04-24T23:42:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.147+0000 7f2574e90640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 0x7f255c0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2560005fd0 tx=0x7f2560005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.151+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f256c0bc5f0 con 0x7f25700ffa50 2024-04-24T23:42:11.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.326+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f257002d170 con 0x7f25700ffa50 2024-04-24T23:42:11.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.326+0000 7f25667fc640 1 -- 172.21.15.107:0/4049065191 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f257002d170 con 0x7f25700ffa50 2024-04-24T23:42:11.326 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:42:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 msgr2=0x7f255c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 0x7f255c0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2560005fd0 tx=0x7f2560005850 comp rx=0 tx=0).stop 2024-04-24T23:42:11.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 msgr2=0x7f2570197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f256c031440 tx=0x7f256c032e20 comp rx=0 tx=0).stop 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 shutdown_connections 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f255c0a2c10 0x7f255c0a50d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25701033e0 0x7f257019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2570102a20 0x7f2570198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 --2- 172.21.15.107:0/4049065191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25700ffa50 0x7f2570197af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 >> 172.21.15.107:0/4049065191 conn(0x7f25700fb380 msgr2=0x7f2570105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.328+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 shutdown_connections 2024-04-24T23:42:11.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.329+0000 7f25778f5640 1 -- 172.21.15.107:0/4049065191 wait complete. 2024-04-24T23:42:11.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 -- 172.21.15.107:0/3128692832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700102250 msgr2=0x7f1700108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 --2- 172.21.15.107:0/3128692832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700102250 0x7f1700108ae0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009ca0 tx=0x7f16f802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 -- 172.21.15.107:0/3128692832 shutdown_connections 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 --2- 172.21.15.107:0/3128692832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700102250 0x7f1700108ae0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 --2- 172.21.15.107:0/3128692832 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700101890 0x7f1700101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 --2- 172.21.15.107:0/3128692832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 0x7f1700100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 -- 172.21.15.107:0/3128692832 >> 172.21.15.107:0/3128692832 conn(0x7f17000fbc30 msgr2=0x7f17000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 -- 172.21.15.107:0/3128692832 shutdown_connections 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.436+0000 7f1706e38640 1 -- 172.21.15.107:0/3128692832 wait complete. 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 Processor -- start 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 -- start start 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 0x7f170006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700102250 0x7f170019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f170010c3c0 con 0x7f1700101890 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f170010c240 con 0x7f17001006a0 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f170010c540 con 0x7f1700102250 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55006/0 (socket says 172.21.15.107:55006) 2024-04-24T23:42:11.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 -- 172.21.15.107:0/3852964596 learned_addr learned my addr 172.21.15.107:0/3852964596 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f17053d5640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700102250 0x7f170019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1704bd4640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 0x7f170006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 -- 172.21.15.107:0/3852964596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700102250 msgr2=0x7f170019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700102250 0x7f170019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 -- 172.21.15.107:0/3852964596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 msgr2=0x7f170006a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 0x7f170006a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 -- 172.21.15.107:0/3852964596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170019f750 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fffff640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00d770 tx=0x7f16ec00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec017070 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ec009820 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec004cd0 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.437+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170019f9e0 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.438+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170019ff60 con 0x7f1700101890 2024-04-24T23:42:11.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.439+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1700107960 con 0x7f1700101890 2024-04-24T23:42:11.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.440+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f16ec0099d0 con 0x7f1700101890 2024-04-24T23:42:11.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.440+0000 7f16fdffb640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:11.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.440+0000 7f1704bd4640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:11.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.440+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f16ec0ce530 con 0x7f1700101890 2024-04-24T23:42:11.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.441+0000 7f1704bd4640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 0x7f16f00a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f17001016f0 tx=0x7f16f4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:11.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.445+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f16ec09b930 con 0x7f1700101890 2024-04-24T23:42:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.659+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f170002d3c0 con 0x7f1700101890 2024-04-24T23:42:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.659+0000 7f16fdffb640 1 -- 172.21.15.107:0/3852964596 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f16ec09b230 con 0x7f1700101890 2024-04-24T23:42:11.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:11.659 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-24T23:42:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 1 2024-04-24T23:42:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 msgr2=0x7f16f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 0x7f16f00a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f17001016f0 tx=0x7f16f4009110 comp rx=0 tx=0).stop 2024-04-24T23:42:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 msgr2=0x7f170019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00d770 tx=0x7f16ec00db70 comp rx=0 tx=0).stop 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 shutdown_connections 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1700102250 0x7f170019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1700101890 0x7f170019c820 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 --2- 172.21.15.107:0/3852964596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17001006a0 0x7f170006a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 >> 172.21.15.107:0/3852964596 conn(0x7f17000fbc30 msgr2=0x7f1700104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 shutdown_connections 2024-04-24T23:42:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:11.661+0000 7f1706e38640 1 -- 172.21.15.107:0/3852964596 wait complete. 2024-04-24T23:42:11.807 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:11 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2449522987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:11.807 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:11 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4049065191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:11.807 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:11 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3852964596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:12.062 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:11 smithi195 systemd[1]: Starting Ceph mds.l for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:12.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:11 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2449522987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:12.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:11 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4049065191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:12.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:11 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3852964596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:12.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:11 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2449522987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:12.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:11 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4049065191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:12.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:11 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3852964596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:12.467 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 podman[62933]: 2024-04-24 23:42:12.11342577 +0000 UTC m=+0.017999608 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:12.467 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 podman[62933]: 2024-04-24 23:42:12.223199892 +0000 UTC m=+0.127773698 container create b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-24T23:42:12.468 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 podman[62933]: 2024-04-24 23:42:12.44262372 +0000 UTC m=+0.347197531 container init b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-24T23:42:12.468 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 podman[62933]: 2024-04-24 23:42:12.445065744 +0000 UTC m=+0.349639558 container start b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T23:42:12.468 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mds[62952]: -- 172.21.15.195:0/305330486 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea1fd67a40 con 0x55ea1fd3a800 2024-04-24T23:42:12.724 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: starting mds.l at 2024-04-24T23:42:12.724 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 bash[62933]: b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 2024-04-24T23:42:12.724 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:42:12 smithi195 systemd[1]: Started Ceph mds.l for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:13.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:13.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6832/1442081594,v1:172.21.15.195:6833/1442081594] up:boot 2024-04-24T23:42:13.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: fsmap 1 up:standby 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:13.063 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:12 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:13.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:13.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6832/1442081594,v1:172.21.15.195:6833/1442081594] up:boot 2024-04-24T23:42:13.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: fsmap 1 up:standby 2024-04-24T23:42:13.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:42:13.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:13.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:13.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:12 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:13.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:13.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6832/1442081594,v1:172.21.15.195:6833/1442081594] up:boot 2024-04-24T23:42:13.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: fsmap 1 up:standby 2024-04-24T23:42:13.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:42:13.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.129 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:13.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:13.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:13.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:12 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:13.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- 172.21.15.107:0/358133392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc102250 msgr2=0x7f15cc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 --2- 172.21.15.107:0/358133392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc102250 0x7f15cc108ae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009ca0 tx=0x7f15c402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- 172.21.15.107:0/358133392 shutdown_connections 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 --2- 172.21.15.107:0/358133392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc102250 0x7f15cc108ae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 --2- 172.21.15.107:0/358133392 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc101890 0x7f15cc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 --2- 172.21.15.107:0/358133392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc1006a0 0x7f15cc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- 172.21.15.107:0/358133392 >> 172.21.15.107:0/358133392 conn(0x7f15cc0fbc30 msgr2=0x7f15cc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- 172.21.15.107:0/358133392 shutdown_connections 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- 172.21.15.107:0/358133392 wait complete. 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 Processor -- start 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.784+0000 7f15d2753640 1 -- start start 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc101890 0x7f15cc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc102250 0x7f15cc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc10d440 con 0x7f15cc1006a0 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc10d2c0 con 0x7f15cc101890 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc10d5c0 con 0x7f15cc102250 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55020/0 (socket says 172.21.15.107:55020) 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 -- 172.21.15.107:0/3306586864 learned_addr learned my addr 172.21.15.107:0/3306586864 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:13.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cb7fe640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc101890 0x7f15cc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d0cf0640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc102250 0x7f15cc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 -- 172.21.15.107:0/3306586864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc102250 msgr2=0x7f15cc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc102250 0x7f15cc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 -- 172.21.15.107:0/3306586864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc101890 msgr2=0x7f15cc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc101890 0x7f15cc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 -- 172.21.15.107:0/3306586864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc19f650 con 0x7f15cc1006a0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15cbfff640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00e9c0 tx=0x7f15bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc00cd80 con 0x7f15cc1006a0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15bc0035f0 con 0x7f15cc1006a0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc19f940 con 0x7f15cc1006a0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.785+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc010690 con 0x7f15cc1006a0 2024-04-24T23:42:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.786+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc19fe90 con 0x7f15cc1006a0 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.786+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15cc107960 con 0x7f15cc1006a0 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.787+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f15bc0037a0 con 0x7f15cc1006a0 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.788+0000 7f15c97fa640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 0x7f15b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.788+0000 7f15cb7fe640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 0x7f15b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.788+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15bc0cef80 con 0x7f15cc1006a0 2024-04-24T23:42:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.789+0000 7f15cb7fe640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 0x7f15b40a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15cc198f40 tx=0x7f15c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:13.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:13.792+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15bc09c380 con 0x7f15cc1006a0 2024-04-24T23:42:13.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:13 smithi170 ceph-mon[32714]: Deploying daemon mds.k on smithi170 2024-04-24T23:42:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.007+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15cc02d170 con 0x7f15cc1006a0 2024-04-24T23:42:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.007+0000 7f15c97fa640 1 -- 172.21.15.107:0/3306586864 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f15bc09bc80 con 0x7f15cc1006a0 2024-04-24T23:42:14.007 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:14.007 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T23:42:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 2 2024-04-24T23:42:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 msgr2=0x7f15b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 0x7f15b40a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15cc198f40 tx=0x7f15c000b390 comp rx=0 tx=0).stop 2024-04-24T23:42:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 msgr2=0x7f15cc197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00e9c0 tx=0x7f15bc00ee80 comp rx=0 tx=0).stop 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 shutdown_connections 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f15b40a2b90 0x7f15b40a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f15cc102250 0x7f15cc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15cc101890 0x7f15cc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 --2- 172.21.15.107:0/3306586864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc1006a0 0x7f15cc197a30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 >> 172.21.15.107:0/3306586864 conn(0x7f15cc0fbc30 msgr2=0x7f15cc104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 shutdown_connections 2024-04-24T23:42:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:14.009+0000 7f15d2753640 1 -- 172.21.15.107:0/3306586864 wait complete. 2024-04-24T23:42:14.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:13 smithi107 ceph-mon[29118]: Deploying daemon mds.k on smithi170 2024-04-24T23:42:14.061 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:13 smithi195 ceph-mon[33019]: Deploying daemon mds.k on smithi170 2024-04-24T23:42:15.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:14 smithi195 ceph-mon[33019]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:15.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:14 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3306586864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:15.069 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:14 smithi170 ceph-mon[32714]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:15.069 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:14 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3306586864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:15.069 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:14 smithi170 systemd[1]: Starting Ceph mds.k for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:15.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:14 smithi107 ceph-mon[29118]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:15.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3306586864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:15.335 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 podman[62091]: 2024-04-24 23:42:14.972525234 +0000 UTC m=+0.018420291 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:15.335 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 podman[62091]: 2024-04-24 23:42:15.090109385 +0000 UTC m=+0.136004441 container create bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-24T23:42:15.835 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 podman[62091]: 2024-04-24 23:42:15.342634382 +0000 UTC m=+0.388529437 container init bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T23:42:15.835 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 podman[62091]: 2024-04-24 23:42:15.345205268 +0000 UTC m=+0.391100325 container start bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-24T23:42:15.836 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 ceph-mds[62110]: -- 172.21.15.170:0/1261724815 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c7cbc3a40 con 0x560c7cb96800 2024-04-24T23:42:15.836 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: starting mds.k at 2024-04-24T23:42:15.836 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 bash[62091]: bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da 2024-04-24T23:42:15.836 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:42:15 smithi170 systemd[1]: Started Ceph mds.k for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:16.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 -- 172.21.15.107:0/2754703351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14102200 msgr2=0x7f0b141045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 --2- 172.21.15.107:0/2754703351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14102200 0x7f0b141045f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009900 tx=0x7f0b0402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 -- 172.21.15.107:0/2754703351 shutdown_connections 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 --2- 172.21.15.107:0/2754703351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b14107460 0x7f0b141098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 --2- 172.21.15.107:0/2754703351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14104b30 0x7f0b14106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 --2- 172.21.15.107:0/2754703351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14102200 0x7f0b141045f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 -- 172.21.15.107:0/2754703351 >> 172.21.15.107:0/2754703351 conn(0x7f0b140fbbf0 msgr2=0x7f0b140fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.131+0000 7f0b19464640 1 -- 172.21.15.107:0/2754703351 shutdown_connections 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 -- 172.21.15.107:0/2754703351 wait complete. 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 Processor -- start 2024-04-24T23:42:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 -- start start 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 0x7f0b14197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b14104b30 0x7f0b14198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1410d4a0 con 0x7f0b14107460 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1410d320 con 0x7f0b14102200 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b19464640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1410d620 con 0x7f0b14104b30 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55040/0 (socket says 172.21.15.107:55040) 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.132+0000 7f0b137fe640 1 -- 172.21.15.107:0/1230254875 learned_addr learned my addr 172.21.15.107:0/1230254875 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 -- 172.21.15.107:0/1230254875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b14104b30 msgr2=0x7f0b14198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b14104b30 0x7f0b14198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 -- 172.21.15.107:0/1230254875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 msgr2=0x7f0b14197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:16.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b12ffd640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 0x7f0b14197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 0x7f0b14197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 -- 172.21.15.107:0/1230254875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1419f760 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b12ffd640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 0x7f0b14197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b137fe640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00cad0 tx=0x7f0b0c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00ae20 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b0c005020 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1419fa50 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00f6f0 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.133+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b141a0020 con 0x7f0b14107460 2024-04-24T23:42:16.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.134+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b140fd6e0 con 0x7f0b14107460 2024-04-24T23:42:16.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.135+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0b0c0051d0 con 0x7f0b14107460 2024-04-24T23:42:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.135+0000 7f0b10ff9640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 0x7f0b000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.136+0000 7f0b12ffd640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 0x7f0b000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.136+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0b0c0cefb0 con 0x7f0b14107460 2024-04-24T23:42:16.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.136+0000 7f0b12ffd640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 0x7f0b000a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b04000c00 tx=0x7f0b04031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:16.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.140+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b0c014030 con 0x7f0b14107460 2024-04-24T23:42:16.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.362+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b1402d370 con 0x7f0b14107460 2024-04-24T23:42:16.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.363+0000 7f0b10ff9640 1 -- 172.21.15.107:0/1230254875 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f0b0c09c3b0 con 0x7f0b14107460 2024-04-24T23:42:16.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:16.363 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:42:16.363 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 msgr2=0x7f0b000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 0x7f0b000a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b04000c00 tx=0x7f0b04031400 comp rx=0 tx=0).stop 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 msgr2=0x7f0b1419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00cad0 tx=0x7f0b0c00cf90 comp rx=0 tx=0).stop 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 shutdown_connections 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b000a2b90 0x7f0b000a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b14107460 0x7f0b1419f0f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b14104b30 0x7f0b14198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 --2- 172.21.15.107:0/1230254875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b14102200 0x7f0b14197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 >> 172.21.15.107:0/1230254875 conn(0x7f0b140fbbf0 msgr2=0x7f0b14102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 shutdown_connections 2024-04-24T23:42:16.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:16.365+0000 7f0b19464640 1 -- 172.21.15.107:0/1230254875 wait complete. 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: Deploying daemon mds.j on smithi107 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6832/3444431424,v1:172.21.15.170:6833/3444431424] up:boot 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: fsmap 2 up:standby 2024-04-24T23:42:16.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:42:16.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:16.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1230254875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:16.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: Deploying daemon mds.j on smithi107 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6832/3444431424,v1:172.21.15.170:6833/3444431424] up:boot 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: fsmap 2 up:standby 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:16.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1230254875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:16.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: Deploying daemon mds.j on smithi107 2024-04-24T23:42:16.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6832/3444431424,v1:172.21.15.170:6833/3444431424] up:boot 2024-04-24T23:42:16.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: fsmap 2 up:standby 2024-04-24T23:42:16.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:42:16.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:16.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1230254875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:17.878 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:17 smithi107 systemd[1]: Starting Ceph mds.j for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:17 smithi107 podman[67377]: 2024-04-24 23:42:17.984657767 +0000 UTC m=+0.108927009 container create 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2) 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:17 smithi107 podman[67377]: 2024-04-24 23:42:17.894000886 +0000 UTC m=+0.018270141 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 podman[67377]: 2024-04-24 23:42:18.204290139 +0000 UTC m=+0.328559381 container init 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 podman[67377]: 2024-04-24 23:42:18.206841374 +0000 UTC m=+0.331110618 container start 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 ceph-mds[67397]: -- 172.21.15.107:0/3767683246 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fef0523a40 con 0x55fef04f6800 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j[67393]: starting mds.j at 2024-04-24T23:42:18.295 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 bash[67377]: 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 2024-04-24T23:42:18.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 -- 172.21.15.107:0/2660360524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40107460 msgr2=0x7f9b401098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2660360524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40107460 0x7f9b401098b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009ca0 tx=0x7f9b3802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 -- 172.21.15.107:0/2660360524 shutdown_connections 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2660360524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40107460 0x7f9b401098b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2660360524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40104b30 0x7f9b40106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2660360524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40102200 0x7f9b401045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 -- 172.21.15.107:0/2660360524 >> 172.21.15.107:0/2660360524 conn(0x7f9b400fbbf0 msgr2=0x7f9b400fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 -- 172.21.15.107:0/2660360524 shutdown_connections 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.482+0000 7f9b4763f640 1 -- 172.21.15.107:0/2660360524 wait complete. 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 Processor -- start 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 -- start start 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40102200 0x7f9b40101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40107460 0x7f9b401006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010d420 con 0x7f9b40104b30 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010d2a0 con 0x7f9b40107460 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b4763f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4010d5a0 con 0x7f9b40102200 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60062/0 (socket says 172.21.15.107:60062) 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 -- 172.21.15.107:0/2868853914 learned_addr learned my addr 172.21.15.107:0/2868853914 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b453db640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40102200 0x7f9b40101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b45bdc640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40107460 0x7f9b401006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 -- 172.21.15.107:0/2868853914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40102200 msgr2=0x7f9b40101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40102200 0x7f9b40101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 -- 172.21.15.107:0/2868853914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40107460 msgr2=0x7f9b401006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40107460 0x7f9b401006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 -- 172.21.15.107:0/2868853914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b40100f80 con 0x7f9b40104b30 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.483+0000 7f9b44bda640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c970 tx=0x7f9b3000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.484+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007bb0 con 0x7f9b40104b30 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.484+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b401a1b60 con 0x7f9b40104b30 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.484+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b30007d50 con 0x7f9b40104b30 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.484+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b300187a0 con 0x7f9b40104b30 2024-04-24T23:42:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.484+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b401a20b0 con 0x7f9b40104b30 2024-04-24T23:42:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.485+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b400fd6e0 con 0x7f9b40104b30 2024-04-24T23:42:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.486+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9b3000f450 con 0x7f9b40104b30 2024-04-24T23:42:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.486+0000 7f9b367fc640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 0x7f9b2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.487+0000 7f9b453db640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 0x7f9b2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.487+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9b300cf050 con 0x7f9b40104b30 2024-04-24T23:42:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.487+0000 7f9b453db640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 0x7f9b2c0a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0058c0 tx=0x7f9b3c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.490+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b30014030 con 0x7f9b40104b30 2024-04-24T23:42:18.627 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:42:18 smithi107 systemd[1]: Started Ceph mds.j for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:18.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.706+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b4002d320 con 0x7f9b40104b30 2024-04-24T23:42:18.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.707+0000 7f9b367fc640 1 -- 172.21.15.107:0/2868853914 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f9b3009c450 con 0x7f9b40104b30 2024-04-24T23:42:18.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:18.707 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:42:18.707 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 msgr2=0x7f9b2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 0x7f9b2c0a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0058c0 tx=0x7f9b3c00a2d0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 msgr2=0x7f9b401001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c970 tx=0x7f9b3000ce30 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 shutdown_connections 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9b2c0a2b90 0x7f9b2c0a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b40107460 0x7f9b401006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b40104b30 0x7f9b401001b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 --2- 172.21.15.107:0/2868853914 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9b40102200 0x7f9b40101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 >> 172.21.15.107:0/2868853914 conn(0x7f9b400fbbf0 msgr2=0x7f9b40102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 shutdown_connections 2024-04-24T23:42:18.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:18.709+0000 7f9b4763f640 1 -- 172.21.15.107:0/2868853914 wait complete. 2024-04-24T23:42:19.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:19.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2868853914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:19.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:19.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2868853914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2868853914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:20.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:20 smithi195 ceph-mon[33019]: Deploying daemon mds.d on smithi107 2024-04-24T23:42:20.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:20 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6834/3058072142,v1:172.21.15.107:6835/3058072142] up:boot 2024-04-24T23:42:20.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:20 smithi195 ceph-mon[33019]: fsmap 3 up:standby 2024-04-24T23:42:20.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T23:42:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:20 smithi170 ceph-mon[32714]: Deploying daemon mds.d on smithi107 2024-04-24T23:42:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:20 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6834/3058072142,v1:172.21.15.107:6835/3058072142] up:boot 2024-04-24T23:42:20.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:20 smithi170 ceph-mon[32714]: fsmap 3 up:standby 2024-04-24T23:42:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T23:42:20.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:20 smithi107 ceph-mon[29118]: Deploying daemon mds.d on smithi107 2024-04-24T23:42:20.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:20 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6834/3058072142,v1:172.21.15.107:6835/3058072142] up:boot 2024-04-24T23:42:20.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:20 smithi107 ceph-mon[29118]: fsmap 3 up:standby 2024-04-24T23:42:20.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T23:42:20.628 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:20 smithi107 systemd[1]: Starting Ceph mds.d for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:20.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 -- 172.21.15.107:0/2471396736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3868102250 msgr2=0x7f3868108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 --2- 172.21.15.107:0/2471396736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3868102250 0x7f3868108ae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3860009ca0 tx=0x7f386002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 -- 172.21.15.107:0/2471396736 shutdown_connections 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 --2- 172.21.15.107:0/2471396736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3868102250 0x7f3868108ae0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 --2- 172.21.15.107:0/2471396736 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868101890 0x7f3868101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 --2- 172.21.15.107:0/2471396736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38681006a0 0x7f3868100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 -- 172.21.15.107:0/2471396736 >> 172.21.15.107:0/2471396736 conn(0x7f38680fbc30 msgr2=0x7f38680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 -- 172.21.15.107:0/2471396736 shutdown_connections 2024-04-24T23:42:20.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.824+0000 7f386cb90640 1 -- 172.21.15.107:0/2471396736 wait complete. 2024-04-24T23:42:20.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 Processor -- start 2024-04-24T23:42:20.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 -- start start 2024-04-24T23:42:20.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:20.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 0x7f386819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868102250 0x7f38681a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f386810c3c0 con 0x7f38681006a0 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f386810c240 con 0x7f3868101890 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386cb90640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f386810c540 con 0x7f3868102250 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60088/0 (socket says 172.21.15.107:60088) 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 -- 172.21.15.107:0/952197566 learned_addr learned my addr 172.21.15.107:0/952197566 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f3866d9d640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868102250 0x7f38681a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f385dd9b640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 0x7f386819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 -- 172.21.15.107:0/952197566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868102250 msgr2=0x7f38681a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868102250 0x7f38681a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 -- 172.21.15.107:0/952197566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 msgr2=0x7f386819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 0x7f386819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:20.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 -- 172.21.15.107:0/952197566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38681a1960 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f385dd9b640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 0x7f386819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.825+0000 7f386659c640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f385800b450 tx=0x7f385800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.826+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858015020 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.826+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38681a1bf0 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.826+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858011d70 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.826+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385800f340 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.826+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38681a2190 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.827+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3868107960 con 0x7f38681006a0 2024-04-24T23:42:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.828+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3858018020 con 0x7f38681006a0 2024-04-24T23:42:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.828+0000 7f385ffff640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 0x7f38540a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.828+0000 7f385dd9b640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 0x7f38540a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.828+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f38580ce600 con 0x7f38681006a0 2024-04-24T23:42:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.829+0000 7f385dd9b640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 0x7f38540a5080 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f386806a680 tx=0x7f3850005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:20.833+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f385801f020 con 0x7f38681006a0 2024-04-24T23:42:21.019 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:20 smithi107 podman[67727]: 2024-04-24 23:42:20.665673553 +0000 UTC m=+0.017645135 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:21.019 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:20 smithi107 podman[67727]: 2024-04-24 23:42:20.792448035 +0000 UTC m=+0.144419590 container create 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-24T23:42:21.019 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 podman[67727]: 2024-04-24 23:42:21.020253401 +0000 UTC m=+0.372224964 container init 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:42:21.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.057+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f386802d3c0 con 0x7f38681006a0 2024-04-24T23:42:21.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.057+0000 7f385ffff640 1 -- 172.21.15.107:0/952197566 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f385800d040 con 0x7f38681006a0 2024-04-24T23:42:21.057 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:21.057 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:42:21.058 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 4 2024-04-24T23:42:21.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 msgr2=0x7f38540a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:21.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 0x7f38540a5080 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f386806a680 tx=0x7f3850005f90 comp rx=0 tx=0).stop 2024-04-24T23:42:21.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 msgr2=0x7f386806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f385800b450 tx=0x7f385800b910 comp rx=0 tx=0).stop 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 shutdown_connections 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38540a2bc0 0x7f38540a5080 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3868102250 0x7f38681a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3868101890 0x7f386819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 --2- 172.21.15.107:0/952197566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38681006a0 0x7f386806a910 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 >> 172.21.15.107:0/952197566 conn(0x7f38680fbc30 msgr2=0x7f3868104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 shutdown_connections 2024-04-24T23:42:21.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:21.059+0000 7f386cb90640 1 -- 172.21.15.107:0/952197566 wait complete. 2024-04-24T23:42:21.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:21 smithi195 ceph-mon[33019]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:21.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:21 smithi170 ceph-mon[32714]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:21.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:21 smithi107 ceph-mon[29118]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:21.378 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 podman[67727]: 2024-04-24 23:42:21.02310544 +0000 UTC m=+0.375077001 container start 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-24T23:42:21.378 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 ceph-mds[67785]: -- 172.21.15.107:0/3041785172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c55e37a40 con 0x564c55e0b000 2024-04-24T23:42:21.378 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d[67781]: starting mds.d at 2024-04-24T23:42:21.378 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 bash[67727]: 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 2024-04-24T23:42:21.378 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:42:21 smithi107 systemd[1]: Started Ceph mds.d for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:22.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/952197566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:22.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:22.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:22 smithi170 ceph-mon[32714]: Deploying daemon mds.e on smithi170 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/952197566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:22.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:22 smithi195 ceph-mon[33019]: Deploying daemon mds.e on smithi170 2024-04-24T23:42:22.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/952197566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:22 smithi107 ceph-mon[29118]: Deploying daemon mds.e on smithi170 2024-04-24T23:42:23.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 -- 172.21.15.107:0/1172795316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c102230 msgr2=0x7fe65c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 --2- 172.21.15.107:0/1172795316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c102230 0x7fe65c108ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe654009ca0 tx=0x7fe65402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 -- 172.21.15.107:0/1172795316 shutdown_connections 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 --2- 172.21.15.107:0/1172795316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c102230 0x7fe65c108ac0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 --2- 172.21.15.107:0/1172795316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c101870 0x7fe65c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 --2- 172.21.15.107:0/1172795316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 -- 172.21.15.107:0/1172795316 >> 172.21.15.107:0/1172795316 conn(0x7fe65c0fbbf0 msgr2=0x7fe65c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 -- 172.21.15.107:0/1172795316 shutdown_connections 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.173+0000 7fe663961640 1 -- 172.21.15.107:0/1172795316 wait complete. 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 Processor -- start 2024-04-24T23:42:23.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 -- start start 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 0x7fe65c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10c360 con 0x7fe65c101870 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10c1e0 con 0x7fe65c100680 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe663961640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10c4e0 con 0x7fe65c102230 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe660efc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe660efc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60134/0 (socket says 172.21.15.107:60134) 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe660efc640 1 -- 172.21.15.107:0/603283937 learned_addr learned my addr 172.21.15.107:0/603283937 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.174+0000 7fe661efe640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 0x7fe65c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 -- 172.21.15.107:0/603283937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 msgr2=0x7fe65c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 0x7fe65c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 -- 172.21.15.107:0/603283937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 msgr2=0x7fe65c069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:23.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe6616fd640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 -- 172.21.15.107:0/603283937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65c19f720 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe661efe640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 0x7fe65c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe6616fd640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe660efc640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00b4a0 tx=0x7fe64c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c014070 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65c19f9b0 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe64c010af0 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c017d10 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.175+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65c19ff80 con 0x7fe65c101870 2024-04-24T23:42:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.176+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65c107940 con 0x7fe65c101870 2024-04-24T23:42:23.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.178+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe64c017070 con 0x7fe65c101870 2024-04-24T23:42:23.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.178+0000 7fe6527fc640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 0x7fe6480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:23.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.179+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe64c0ce090 con 0x7fe65c101870 2024-04-24T23:42:23.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.179+0000 7fe6616fd640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 0x7fe6480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:23.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.179+0000 7fe6616fd640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 0x7fe6480a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe65c1016d0 tx=0x7fe658005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.183+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe64c09b490 con 0x7fe65c101870 2024-04-24T23:42:23.251 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:23 smithi170 ceph-mon[32714]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:23.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:23 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6836/3538832535,v1:172.21.15.107:6837/3538832535] up:boot 2024-04-24T23:42:23.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:23 smithi170 ceph-mon[32714]: fsmap 4 up:standby 2024-04-24T23:42:23.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T23:42:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:23 smithi195 ceph-mon[33019]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:23 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6836/3538832535,v1:172.21.15.107:6837/3538832535] up:boot 2024-04-24T23:42:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:23 smithi195 ceph-mon[33019]: fsmap 4 up:standby 2024-04-24T23:42:23.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T23:42:23.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:23 smithi107 ceph-mon[29118]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:23.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:23 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6836/3538832535,v1:172.21.15.107:6837/3538832535] up:boot 2024-04-24T23:42:23.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:23 smithi107 ceph-mon[29118]: fsmap 4 up:standby 2024-04-24T23:42:23.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T23:42:23.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.403+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe65c02d340 con 0x7fe65c101870 2024-04-24T23:42:23.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.404+0000 7fe6527fc640 1 -- 172.21.15.107:0/603283937 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7fe64c09ad90 con 0x7fe65c101870 2024-04-24T23:42:23.458 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:23.458 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 5 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 msgr2=0x7fe6480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 0x7fe6480a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe65c1016d0 tx=0x7fe658005d30 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 msgr2=0x7fe65c19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00b4a0 tx=0x7fe64c00b960 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 shutdown_connections 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe6480a2b90 0x7fe6480a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe65c102230 0x7fe65c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c101870 0x7fe65c19c7f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 --2- 172.21.15.107:0/603283937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe65c100680 0x7fe65c069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 >> 172.21.15.107:0/603283937 conn(0x7fe65c0fbbf0 msgr2=0x7fe65c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 shutdown_connections 2024-04-24T23:42:23.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:23.406+0000 7fe663961640 1 -- 172.21.15.107:0/603283937 wait complete. 2024-04-24T23:42:24.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:24 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/603283937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:24.283 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:42:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: starting mds.e at 2024-04-24T23:42:24.315 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:24 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/603283937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:24.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:24 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/603283937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:24.585 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:42:24 smithi170 bash[62415]: ea811e32727d137102c8f172e7ec87873c6b825943b2c5c5041be330c12ad73c 2024-04-24T23:42:24.586 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:42:24 smithi170 systemd[1]: Started Ceph mds.e for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:25.264 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:25.264 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.265 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.265 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.265 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:25.265 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:25.265 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:25.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:25.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:25.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:25.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:25.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:25.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/720311844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44101880 msgr2=0x7f8e44101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/720311844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44101880 0x7f8e44101d00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009900 tx=0x7f8e3402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/720311844 shutdown_connections 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/720311844 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44102240 0x7f8e44108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/720311844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44101880 0x7f8e44101d00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/720311844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44100690 0x7f8e44100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/720311844 >> 172.21.15.107:0/720311844 conn(0x7f8e440fbc20 msgr2=0x7f8e440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/720311844 shutdown_connections 2024-04-24T23:42:25.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/720311844 wait complete. 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.574+0000 7f8e4b1c1640 1 Processor -- start 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 -- start start 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44101880 0x7f8e44197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 0x7f8e4419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4410d430 con 0x7f8e44100690 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4410d2b0 con 0x7f8e44102240 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4410d5b0 con 0x7f8e44101880 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60146/0 (socket says 172.21.15.107:60146) 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 -- 172.21.15.107:0/1358151723 learned_addr learned my addr 172.21.15.107:0/1358151723 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e43fff640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44101880 0x7f8e44197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:25.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4975e640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 0x7f8e4419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 -- 172.21.15.107:0/1358151723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44101880 msgr2=0x7f8e44197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44101880 0x7f8e44197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 -- 172.21.15.107:0/1358151723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 msgr2=0x7f8e4419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 0x7f8e4419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 -- 172.21.15.107:0/1358151723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4419f620 con 0x7f8e44100690 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4975e640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 0x7f8e4419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e48f5d640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00ccb0 tx=0x7f8e3c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003b70 con 0x7f8e44100690 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e3c00ae50 con 0x7f8e44100690 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4419f910 con 0x7f8e44100690 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.575+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c00f810 con 0x7f8e44100690 2024-04-24T23:42:25.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.576+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4419fe80 con 0x7f8e44100690 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.577+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e441079b0 con 0x7f8e44100690 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.578+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8e3c00f9b0 con 0x7f8e44100690 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.578+0000 7f8e41ffb640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 0x7f8e300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.579+0000 7f8e43fff640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 0x7f8e300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.579+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8e3c0ced10 con 0x7f8e44100690 2024-04-24T23:42:25.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.579+0000 7f8e43fff640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 0x7f8e300a50d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8e340314e0 tx=0x7f8e34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:25.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.583+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e3c014070 con 0x7f8e44100690 2024-04-24T23:42:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.795+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e4402d170 con 0x7f8e44100690 2024-04-24T23:42:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.795+0000 7f8e41ffb640 1 -- 172.21.15.107:0/1358151723 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f8e3c09c0b0 con 0x7f8e44100690 2024-04-24T23:42:25.795 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:25.796 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-24T23:42:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 6 2024-04-24T23:42:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 msgr2=0x7f8e300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 0x7f8e300a50d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8e340314e0 tx=0x7f8e34000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 msgr2=0x7f8e44197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00ccb0 tx=0x7f8e3c00a620 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 shutdown_connections 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8e300a2c10 0x7f8e300a50d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.797+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e44102240 0x7f8e4419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.798+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8e44101880 0x7f8e44197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.798+0000 7f8e4b1c1640 1 --2- 172.21.15.107:0/1358151723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e44100690 0x7f8e44197a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.798+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 >> 172.21.15.107:0/1358151723 conn(0x7f8e440fbc20 msgr2=0x7f8e44104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.798+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 shutdown_connections 2024-04-24T23:42:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:25.798+0000 7f8e4b1c1640 1 -- 172.21.15.107:0/1358151723 wait complete. 2024-04-24T23:42:26.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:26 smithi195 ceph-mon[33019]: Deploying daemon mds.f on smithi195 2024-04-24T23:42:26.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:26 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6834/3179503985,v1:172.21.15.170:6835/3179503985] up:boot 2024-04-24T23:42:26.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:26 smithi195 ceph-mon[33019]: fsmap 5 up:standby 2024-04-24T23:42:26.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:42:26.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:26 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1358151723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:26.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:26 smithi170 ceph-mon[32714]: Deploying daemon mds.f on smithi195 2024-04-24T23:42:26.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:26 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6834/3179503985,v1:172.21.15.170:6835/3179503985] up:boot 2024-04-24T23:42:26.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:26 smithi170 ceph-mon[32714]: fsmap 5 up:standby 2024-04-24T23:42:26.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:42:26.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:26 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1358151723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:26.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:26 smithi107 ceph-mon[29118]: Deploying daemon mds.f on smithi195 2024-04-24T23:42:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:26 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6834/3179503985,v1:172.21.15.170:6835/3179503985] up:boot 2024-04-24T23:42:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:26 smithi107 ceph-mon[29118]: fsmap 5 up:standby 2024-04-24T23:42:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:42:26.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:26 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1358151723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:26.761 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:26 smithi195 systemd[1]: Starting Ceph mds.f for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:27.021 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:26 smithi195 podman[63247]: 2024-04-24 23:42:26.664709802 +0000 UTC m=+0.017660885 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:27.021 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:26 smithi195 podman[63247]: 2024-04-24 23:42:26.778182375 +0000 UTC m=+0.131133447 container create ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T23:42:27.021 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 podman[63247]: 2024-04-24 23:42:27.022583404 +0000 UTC m=+0.375534477 container init ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:27 smithi195 ceph-mon[33019]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 podman[63247]: 2024-04-24 23:42:27.025292236 +0000 UTC m=+0.378243311 container start ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 ceph-mds[63266]: -- 172.21.15.195:0/3906126223 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3910a5a40 con 0x55c391078c00 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: starting mds.f at 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 bash[63247]: ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf 2024-04-24T23:42:27.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:42:27 smithi195 systemd[1]: Started Ceph mds.f for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:27 smithi170 ceph-mon[32714]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:27.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:27 smithi107 ceph-mon[29118]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:27.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:27.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 -- 172.21.15.107:0/2437221476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 msgr2=0x7f0030102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:27.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 --2- 172.21.15.107:0/2437221476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f0030102440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0024009900 tx=0x7f002402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:27.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 -- 172.21.15.107:0/2437221476 shutdown_connections 2024-04-24T23:42:27.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 --2- 172.21.15.107:0/2437221476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 0x7f00301057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:27.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 --2- 172.21.15.107:0/2437221476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 0x7f0030104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 --2- 172.21.15.107:0/2437221476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f0030102440 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.918+0000 7f0036d4f640 1 -- 172.21.15.107:0/2437221476 >> 172.21.15.107:0/2437221476 conn(0x7f00300fbc30 msgr2=0x7f00300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- 172.21.15.107:0/2437221476 shutdown_connections 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- 172.21.15.107:0/2437221476 wait complete. 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 Processor -- start 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- start start 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 0x7f003019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 0x7f00301a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0030113a10 con 0x7f0030100010 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0030113890 con 0x7f0030105330 2024-04-24T23:42:27.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0036d4f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0030113b90 con 0x7f0030102980 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0034aeb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0034aeb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44288/0 (socket says 172.21.15.107:44288) 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f0034aeb640 1 -- 172.21.15.107:0/262903106 learned_addr learned my addr 172.21.15.107:0/262903106 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.919+0000 7f002ffff640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 0x7f003019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 -- 172.21.15.107:0/262903106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 msgr2=0x7f003019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 0x7f003019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 -- 172.21.15.107:0/262903106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 msgr2=0x7f00301a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f00352ec640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 0x7f00301a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 0x7f00301a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 -- 172.21.15.107:0/262903106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00301a5ce0 con 0x7f0030100010 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0034aeb640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0024009230 tx=0x7f0024031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002403f030 con 0x7f0030100010 2024-04-24T23:42:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024003ad0 con 0x7f0030100010 2024-04-24T23:42:27.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024035070 con 0x7f0030100010 2024-04-24T23:42:27.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00301a5f70 con 0x7f0030100010 2024-04-24T23:42:27.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.920+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00301a6370 con 0x7f0030100010 2024-04-24T23:42:27.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.921+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00300fd720 con 0x7f0030100010 2024-04-24T23:42:27.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.922+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0024042030 con 0x7f0030100010 2024-04-24T23:42:27.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.923+0000 7f002dffb640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 0x7f00200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:27.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.923+0000 7f002ffff640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 0x7f00200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:27.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.923+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f00240ef180 con 0x7f0030100010 2024-04-24T23:42:27.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.924+0000 7f002ffff640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 0x7f00200a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f003019f5d0 tx=0x7f001c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:27.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:27.927+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00240bc520 con 0x7f0030100010 2024-04-24T23:42:28.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:boot 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: fsmap 6 up:standby 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:28.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:28 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:28.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.143+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f003002d3d0 con 0x7f0030100010 2024-04-24T23:42:28.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.144+0000 7f002dffb640 1 -- 172.21.15.107:0/262903106 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5536 (secure 0 0 0) 0x7f00240bbe20 con 0x7f0030100010 2024-04-24T23:42:28.144 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:28.144 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-24T23:42:28.144 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2024-04-24T23:42:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 msgr2=0x7f00200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 0x7f00200a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f003019f5d0 tx=0x7f001c005f90 comp rx=0 tx=0).stop 2024-04-24T23:42:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 msgr2=0x7f003019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0024009230 tx=0x7f0024031b60 comp rx=0 tx=0).stop 2024-04-24T23:42:28.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 shutdown_connections 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f00200a2b90 0x7f00200a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0030105330 0x7f00301a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0030102980 0x7f003019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 --2- 172.21.15.107:0/262903106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0030100010 0x7f003019e0c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 >> 172.21.15.107:0/262903106 conn(0x7f00300fbc30 msgr2=0x7f0030103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 shutdown_connections 2024-04-24T23:42:28.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:28.146+0000 7f0036d4f640 1 -- 172.21.15.107:0/262903106 wait complete. 2024-04-24T23:42:28.192 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:boot 2024-04-24T23:42:28.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: fsmap 6 up:standby 2024-04-24T23:42:28.193 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:28.194 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:28 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:28.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:boot 2024-04-24T23:42:28.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: fsmap 6 up:standby 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:28.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:28 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:29.188 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:29 smithi195 ceph-mon[33019]: Deploying daemon mds.c on smithi195 2024-04-24T23:42:29.189 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:29 smithi195 ceph-mon[33019]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:29.189 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:29 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/262903106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:29 smithi170 ceph-mon[32714]: Deploying daemon mds.c on smithi195 2024-04-24T23:42:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:29 smithi170 ceph-mon[32714]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:29 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/262903106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:29.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:29 smithi107 ceph-mon[29118]: Deploying daemon mds.c on smithi195 2024-04-24T23:42:29.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:29 smithi107 ceph-mon[29118]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:29.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:29 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/262903106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:29.562 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 systemd[1]: Starting Ceph mds.c for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:29.943 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 podman[63551]: 2024-04-24 23:42:29.486466401 +0000 UTC m=+0.018174835 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:29.944 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 podman[63551]: 2024-04-24 23:42:29.608766224 +0000 UTC m=+0.140474680 container create c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:42:29.944 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 podman[63551]: 2024-04-24 23:42:29.944755105 +0000 UTC m=+0.476463515 container init c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-24T23:42:30.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 -- 172.21.15.107:0/4271150827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62901029a0 msgr2=0x7f6290102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 --2- 172.21.15.107:0/4271150827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62901029a0 0x7f6290102e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6278009900 tx=0x7f627802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 -- 172.21.15.107:0/4271150827 shutdown_connections 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 --2- 172.21.15.107:0/4271150827 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6290103360 0x7f6290109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 --2- 172.21.15.107:0/4271150827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62901029a0 0x7f6290102e20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 --2- 172.21.15.107:0/4271150827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901017b0 0x7f6290101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 -- 172.21.15.107:0/4271150827 >> 172.21.15.107:0/4271150827 conn(0x7f62900fcd40 msgr2=0x7f62900ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 -- 172.21.15.107:0/4271150827 shutdown_connections 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.264+0000 7f6295e76640 1 -- 172.21.15.107:0/4271150827 wait complete. 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 Processor -- start 2024-04-24T23:42:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 -- start start 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62901017b0 0x7f6290197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901029a0 0x7f6290197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f629010d430 con 0x7f6290103360 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f629010d2b0 con 0x7f62901029a0 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f6295e76640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f629010d5b0 con 0x7f62901017b0 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f628ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62901017b0 0x7f6290197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f628ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44298/0 (socket says 172.21.15.107:44298) 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.265+0000 7f628ffff640 1 -- 172.21.15.107:0/3675748461 learned_addr learned my addr 172.21.15.107:0/3675748461 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628effd640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901029a0 0x7f6290197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 -- 172.21.15.107:0/3675748461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62901017b0 msgr2=0x7f6290197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62901017b0 0x7f6290197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 -- 172.21.15.107:0/3675748461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901029a0 msgr2=0x7f6290197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901029a0 0x7f6290197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 -- 172.21.15.107:0/3675748461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629019f6a0 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628ffff640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f628800eaa0 tx=0x7f628800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628800cdb0 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6288005020 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629019f990 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880116b0 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.266+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629019feb0 con 0x7f6290103360 2024-04-24T23:42:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.267+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6290108ad0 con 0x7f6290103360 2024-04-24T23:42:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.268+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6288005560 con 0x7f6290103360 2024-04-24T23:42:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.269+0000 7f628cff9640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 0x7f627c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.269+0000 7f628f7fe640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 0x7f627c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.269+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f62880ced60 con 0x7f6290103360 2024-04-24T23:42:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.269+0000 7f628f7fe640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 0x7f627c0a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6290102800 tx=0x7f628000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.273+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6288016030 con 0x7f6290103360 2024-04-24T23:42:30.312 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 podman[63551]: 2024-04-24 23:42:29.947379039 +0000 UTC m=+0.479087450 container start c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-24T23:42:30.312 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 ceph-mds[63571]: -- 172.21.15.195:0/2625406919 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562329a7da40 con 0x562329a51000 2024-04-24T23:42:30.312 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: starting mds.c at 2024-04-24T23:42:30.312 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:30 smithi195 bash[63551]: c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 2024-04-24T23:42:30.312 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:42:30 smithi195 systemd[1]: Started Ceph mds.c for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:30.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.487+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f629002d170 con 0x7f6290103360 2024-04-24T23:42:30.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.487+0000 7f628cff9640 1 -- 172.21.15.107:0/3675748461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6364 (secure 0 0 0) 0x7f628809bb10 con 0x7f6290103360 2024-04-24T23:42:30.488 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:30.488 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-24T23:42:30.488 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 msgr2=0x7f627c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 0x7f627c0a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6290102800 tx=0x7f628000b390 comp rx=0 tx=0).stop 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 msgr2=0x7f629019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f628800eaa0 tx=0x7f628800ef60 comp rx=0 tx=0).stop 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 shutdown_connections 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f627c0a2b90 0x7f627c0a5050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6290103360 0x7f629019f030 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62901029a0 0x7f6290197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 --2- 172.21.15.107:0/3675748461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f62901017b0 0x7f6290197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.489+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 >> 172.21.15.107:0/3675748461 conn(0x7f62900fcd40 msgr2=0x7f6290105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.490+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 shutdown_connections 2024-04-24T23:42:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:30.490+0000 7f6295e76640 1 -- 172.21.15.107:0/3675748461 wait complete. 2024-04-24T23:42:31.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:31.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:boot 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: fsmap 7 up:standby 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: Deploying daemon mds.h on smithi170 2024-04-24T23:42:31.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:31 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3675748461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:31.372 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:boot 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: fsmap 7 up:standby 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: Deploying daemon mds.h on smithi170 2024-04-24T23:42:31.373 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:31 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3675748461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:boot 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: fsmap 7 up:standby 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:31.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:31.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: Deploying daemon mds.h on smithi170 2024-04-24T23:42:31.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:31 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3675748461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:32.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:32.532 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 systemd[1]: Starting Ceph mds.h for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 -- 172.21.15.107:0/4136809286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04100690 msgr2=0x7fbf04100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 --2- 172.21.15.107:0/4136809286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04100690 0x7fbf04100ab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009900 tx=0x7fbef802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 -- 172.21.15.107:0/4136809286 shutdown_connections 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 --2- 172.21.15.107:0/4136809286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04102240 0x7fbf04108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 --2- 172.21.15.107:0/4136809286 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04101880 0x7fbf04101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 --2- 172.21.15.107:0/4136809286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04100690 0x7fbf04100ab0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 -- 172.21.15.107:0/4136809286 >> 172.21.15.107:0/4136809286 conn(0x7fbf040fbc20 msgr2=0x7fbf040fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 -- 172.21.15.107:0/4136809286 shutdown_connections 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 -- 172.21.15.107:0/4136809286 wait complete. 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.610+0000 7fbf09664640 1 Processor -- start 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 -- start start 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04100690 0x7fbf0406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04101880 0x7fbf0419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0410c310 con 0x7fbf04102240 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0410c190 con 0x7fbf04101880 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf09664640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0410c490 con 0x7fbf04100690 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44322/0 (socket says 172.21.15.107:44322) 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 -- 172.21.15.107:0/306636260 learned_addr learned my addr 172.21.15.107:0/306636260 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf02ffd640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04100690 0x7fbf0406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf027fc640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04101880 0x7fbf0419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 -- 172.21.15.107:0/306636260 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04100690 msgr2=0x7fbf0406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04100690 0x7fbf0406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 -- 172.21.15.107:0/306636260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04101880 msgr2=0x7fbf0419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04101880 0x7fbf0419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 -- 172.21.15.107:0/306636260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0419f760 con 0x7fbf04102240 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbf037fe640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00eaa0 tx=0x7fbefc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00cdb0 con 0x7fbf04102240 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.611+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbefc005020 con 0x7fbf04102240 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.612+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0419f9f0 con 0x7fbf04102240 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.612+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0116b0 con 0x7fbf04102240 2024-04-24T23:42:32.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.612+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0419fef0 con 0x7fbf04102240 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.613+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf04107a00 con 0x7fbf04102240 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.614+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbefc005560 con 0x7fbf04102240 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.614+0000 7fbeebfff640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 0x7fbeec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.614+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbefc0ced40 con 0x7fbf04102240 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.614+0000 7fbf02ffd640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 0x7fbeec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:32.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.615+0000 7fbf02ffd640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 0x7fbeec0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbef8005ab0 tx=0x7fbef8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:32.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.619+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbefc016030 con 0x7fbf04102240 2024-04-24T23:42:32.813 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 podman[62726]: 2024-04-24 23:42:32.435887585 +0000 UTC m=+0.018201168 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:32.813 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 podman[62726]: 2024-04-24 23:42:32.536351714 +0000 UTC m=+0.118665285 container create dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-24T23:42:32.813 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 podman[62726]: 2024-04-24 23:42:32.814461369 +0000 UTC m=+0.396774939 container init dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-24T23:42:32.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.833+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf0402d3c0 con 0x7fbf04102240 2024-04-24T23:42:32.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.833+0000 7fbeebfff640 1 -- 172.21.15.107:0/306636260 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6364 (secure 0 0 0) 0x7fbefc09baf0 con 0x7fbf04102240 2024-04-24T23:42:32.833 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:32.833 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-24T23:42:32.834 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 msgr2=0x7fbeec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 0x7fbeec0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbef8005ab0 tx=0x7fbef8000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 msgr2=0x7fbf0419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00eaa0 tx=0x7fbefc00ef60 comp rx=0 tx=0).stop 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 shutdown_connections 2024-04-24T23:42:32.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbeec0a2b90 0x7fbeec0a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf04102240 0x7fbf0419ee50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf04101880 0x7fbf0419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 --2- 172.21.15.107:0/306636260 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf04100690 0x7fbf0406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 >> 172.21.15.107:0/306636260 conn(0x7fbf040fbc20 msgr2=0x7fbf04104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 shutdown_connections 2024-04-24T23:42:32.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:32.836+0000 7fbf09664640 1 -- 172.21.15.107:0/306636260 wait complete. 2024-04-24T23:42:33.085 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 podman[62726]: 2024-04-24 23:42:32.817168788 +0000 UTC m=+0.399482367 container start dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-24T23:42:33.085 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 ceph-mds[62746]: -- 172.21.15.170:0/883732633 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619965fba40 con 0x5619965cec00 2024-04-24T23:42:33.086 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: starting mds.h at 2024-04-24T23:42:33.086 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 bash[62726]: dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 2024-04-24T23:42:33.086 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:42:32 smithi170 systemd[1]: Started Ceph mds.h for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:33.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/306636260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:33.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:33 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/306636260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:33 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/306636260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:33.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:33 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:34 smithi195 ceph-mon[33019]: Deploying daemon mds.i on smithi195 2024-04-24T23:42:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:34 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:boot 2024-04-24T23:42:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:34 smithi195 ceph-mon[33019]: fsmap 8 up:standby 2024-04-24T23:42:34.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:34 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:42:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:34 smithi107 ceph-mon[29118]: Deploying daemon mds.i on smithi195 2024-04-24T23:42:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:34 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:boot 2024-04-24T23:42:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:34 smithi107 ceph-mon[29118]: fsmap 8 up:standby 2024-04-24T23:42:34.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:34 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:42:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:34 smithi170 ceph-mon[32714]: Deploying daemon mds.i on smithi195 2024-04-24T23:42:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:34 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:boot 2024-04-24T23:42:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:34 smithi170 ceph-mon[32714]: fsmap 8 up:standby 2024-04-24T23:42:34.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:34 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:42:34.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:34.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- 172.21.15.107:0/3944216834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc100680 msgr2=0x7fd3fc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 --2- 172.21.15.107:0/3944216834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc100680 0x7fd3fc100aa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009900 tx=0x7fd3f802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- 172.21.15.107:0/3944216834 shutdown_connections 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 --2- 172.21.15.107:0/3944216834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc102230 0x7fd3fc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 --2- 172.21.15.107:0/3944216834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc101870 0x7fd3fc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 --2- 172.21.15.107:0/3944216834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc100680 0x7fd3fc100aa0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- 172.21.15.107:0/3944216834 >> 172.21.15.107:0/3944216834 conn(0x7fd3fc0fbbf0 msgr2=0x7fd3fc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- 172.21.15.107:0/3944216834 shutdown_connections 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- 172.21.15.107:0/3944216834 wait complete. 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 Processor -- start 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.959+0000 7fd4039db640 1 -- start start 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc100680 0x7fd3fc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc101870 0x7fd3fc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc10c300 con 0x7fd3fc102230 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc10c180 con 0x7fd3fc101870 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc10c480 con 0x7fd3fc100680 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44348/0 (socket says 172.21.15.107:44348) 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 -- 172.21.15.107:0/274903589 learned_addr learned my addr 172.21.15.107:0/274903589 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401777640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc100680 0x7fd3fc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd400f76640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc101870 0x7fd3fc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 -- 172.21.15.107:0/274903589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc100680 msgr2=0x7fd3fc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc100680 0x7fd3fc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 -- 172.21.15.107:0/274903589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc101870 msgr2=0x7fd3fc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc101870 0x7fd3fc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 -- 172.21.15.107:0/274903589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc19f6f0 con 0x7fd3fc102230 2024-04-24T23:42:34.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd401f78640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400cce0 tx=0x7fd3f400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:34.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f400f450 con 0x7fd3fc102230 2024-04-24T23:42:34.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc19f9e0 con 0x7fd3fc102230 2024-04-24T23:42:34.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f400fad0 con 0x7fd3fc102230 2024-04-24T23:42:34.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4018c10 con 0x7fd3fc102230 2024-04-24T23:42:34.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.960+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3fc19fee0 con 0x7fd3fc102230 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.961+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3fc1a0170 con 0x7fd3fc102230 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.963+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd3f400f5f0 con 0x7fd3fc102230 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.963+0000 7fd3ee7fc640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 0x7fd3e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.963+0000 7fd401777640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 0x7fd3e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.963+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd3f40cf260 con 0x7fd3fc102230 2024-04-24T23:42:34.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.964+0000 7fd401777640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 0x7fd3e80a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8005ab0 tx=0x7fd3f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:34.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:34.967+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd3f4014030 con 0x7fd3fc102230 2024-04-24T23:42:35.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:35 smithi107 ceph-mon[29118]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:35.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.181+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3fc1a0340 con 0x7fd3fc102230 2024-04-24T23:42:35.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.181+0000 7fd3ee7fc640 1 -- 172.21.15.107:0/274903589 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7189 (secure 0 0 0) 0x7fd3f409c6d0 con 0x7fd3fc102230 2024-04-24T23:42:35.181 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:35.181 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-24T23:42:35.182 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 9 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 msgr2=0x7fd3e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 0x7fd3e80a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8005ab0 tx=0x7fd3f8000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 msgr2=0x7fd3fc19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400cce0 tx=0x7fd3f400a620 comp rx=0 tx=0).stop 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 shutdown_connections 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3e80a2b90 0x7fd3e80a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:35.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3fc102230 0x7fd3fc19f080 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:35.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc101870 0x7fd3fc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:35.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 --2- 172.21.15.107:0/274903589 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3fc100680 0x7fd3fc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:35.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 >> 172.21.15.107:0/274903589 conn(0x7fd3fc0fbbf0 msgr2=0x7fd3fc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:35.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 shutdown_connections 2024-04-24T23:42:35.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:35.183+0000 7fd4039db640 1 -- 172.21.15.107:0/274903589 wait complete. 2024-04-24T23:42:35.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:35 smithi195 ceph-mon[33019]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:35.312 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 systemd[1]: Starting Ceph mds.i for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:35 smithi170 ceph-mon[32714]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:35.682 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 podman[63871]: 2024-04-24 23:42:35.333767086 +0000 UTC m=+0.017827509 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:35.682 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 podman[63871]: 2024-04-24 23:42:35.438672764 +0000 UTC m=+0.122733178 container create ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 podman[63871]: 2024-04-24 23:42:35.683300688 +0000 UTC m=+0.367361102 container init ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 podman[63871]: 2024-04-24 23:42:35.686248236 +0000 UTC m=+0.370308663 container start ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 ceph-mds[63890]: -- 172.21.15.195:0/3750774534 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d642e1fa40 con 0x55d642df2c00 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: starting mds.i at 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 bash[63871]: ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 2024-04-24T23:42:36.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:42:35 smithi195 systemd[1]: Started Ceph mds.i for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:36.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/274903589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:36.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:36.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:36.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:36 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/274903589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:36 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:36.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/274903589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:36.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:36 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:37.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 -- 172.21.15.107:0/3225998377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 msgr2=0x7f88d4108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3225998377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d4108ad0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009900 tx=0x7f88c402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 -- 172.21.15.107:0/3225998377 shutdown_connections 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3225998377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d4108ad0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3225998377 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 0x7f88d4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3225998377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 0x7f88d4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 -- 172.21.15.107:0/3225998377 >> 172.21.15.107:0/3225998377 conn(0x7f88d40fbc20 msgr2=0x7f88d40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 -- 172.21.15.107:0/3225998377 shutdown_connections 2024-04-24T23:42:37.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.296+0000 7f88d98bd640 1 -- 172.21.15.107:0/3225998377 wait complete. 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 Processor -- start 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 -- start start 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 0x7f88d4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 0x7f88d4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88d410c2f0 con 0x7f88d4102240 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f88d410c170 con 0x7f88d4100690 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d98bd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f88d410c470 con 0x7f88d4101880 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58764/0 (socket says 172.21.15.107:58764) 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.297+0000 7f88d37fe640 1 -- 172.21.15.107:0/3216256300 learned_addr learned my addr 172.21.15.107:0/3216256300 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d27fc640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 0x7f88d4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 -- 172.21.15.107:0/3216256300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 msgr2=0x7f88d4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 0x7f88d4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 -- 172.21.15.107:0/3216256300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 msgr2=0x7f88d4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d2ffd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 0x7f88d4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 0x7f88d4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 -- 172.21.15.107:0/3216256300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d419f6a0 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d37fe640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88c40314e0 tx=0x7f88c4004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c403f030 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c4003ce0 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d419f930 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4035070 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.298+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d419fe70 con 0x7f88d4102240 2024-04-24T23:42:37.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.299+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d41a00a0 con 0x7f88d4102240 2024-04-24T23:42:37.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.301+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f88c4042030 con 0x7f88d4102240 2024-04-24T23:42:37.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.301+0000 7f88d88bb640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 0x7f88bc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:37.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.302+0000 7f88d2ffd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 0x7f88bc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:37.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.302+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f88c40ef5f0 con 0x7f88d4102240 2024-04-24T23:42:37.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.302+0000 7f88d2ffd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 0x7f88bc0a50d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f88cc005fd0 tx=0x7f88cc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:37.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.305+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f88c40bc9f0 con 0x7f88d4102240 2024-04-24T23:42:37.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:37 smithi107 ceph-mon[29118]: Deploying daemon mds.g on smithi107 2024-04-24T23:42:37.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:37 smithi107 ceph-mon[29118]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:37 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:boot 2024-04-24T23:42:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:37 smithi107 ceph-mon[29118]: fsmap 9 up:standby 2024-04-24T23:42:37.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:37 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:42:37.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:37 smithi170 ceph-mon[32714]: Deploying daemon mds.g on smithi107 2024-04-24T23:42:37.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:37 smithi170 ceph-mon[32714]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:37.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:37 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:boot 2024-04-24T23:42:37.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:37 smithi170 ceph-mon[32714]: fsmap 9 up:standby 2024-04-24T23:42:37.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:37 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:42:37.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.522+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d41a0270 con 0x7f88d4102240 2024-04-24T23:42:37.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.522+0000 7f88d88bb640 1 -- 172.21.15.107:0/3216256300 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8019 (secure 0 0 0) 0x7f88c40bc2f0 con 0x7f88d4102240 2024-04-24T23:42:37.522 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:37.522 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T23:42:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 10 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 msgr2=0x7f88bc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 0x7f88bc0a50d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f88cc005fd0 tx=0x7f88cc005850 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 msgr2=0x7f88d419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88c40314e0 tx=0x7f88c4004360 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 shutdown_connections 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88bc0a2c10 0x7f88bc0a50d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88d4102240 0x7f88d419f030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88d4101880 0x7f88d4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 --2- 172.21.15.107:0/3216256300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d4100690 0x7f88d4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 >> 172.21.15.107:0/3216256300 conn(0x7f88d40fbc20 msgr2=0x7f88d40fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 shutdown_connections 2024-04-24T23:42:37.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:37.525+0000 7f88d98bd640 1 -- 172.21.15.107:0/3216256300 wait complete. 2024-04-24T23:42:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:37 smithi195 ceph-mon[33019]: Deploying daemon mds.g on smithi107 2024-04-24T23:42:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:37 smithi195 ceph-mon[33019]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:37 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:boot 2024-04-24T23:42:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:37 smithi195 ceph-mon[33019]: fsmap 9 up:standby 2024-04-24T23:42:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:37 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:42:38.107 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 systemd[1]: Starting Ceph mds.g for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:42:38.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:38 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3216256300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:38.375 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 podman[68376]: 2024-04-24 23:42:38.14966813 +0000 UTC m=+0.017725315 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:42:38.375 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 podman[68376]: 2024-04-24 23:42:38.257737409 +0000 UTC m=+0.125794625 container create cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:42:38.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:38 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3216256300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:38.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:38 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3216256300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:38.628 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 podman[68376]: 2024-04-24 23:42:38.510954603 +0000 UTC m=+0.379011777 container init cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True) 2024-04-24T23:42:38.628 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 podman[68376]: 2024-04-24 23:42:38.51370643 +0000 UTC m=+0.381763617 container start cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-24T23:42:38.628 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 ceph-mds[68395]: -- 172.21.15.107:0/982279518 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56518f357a40 con 0x56518f32a800 2024-04-24T23:42:38.628 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: starting mds.g at 2024-04-24T23:42:39.110 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 bash[68376]: cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 2024-04-24T23:42:39.110 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:42:38 smithi107 systemd[1]: Started Ceph mds.g for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:39.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:39 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:39.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:39 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:39.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:39.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:39.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:39 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:39.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 -- 172.21.15.107:0/2359466245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244101880 msgr2=0x7f8244101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2359466245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244101880 0x7f8244101d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8238009900 tx=0x7f823802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 -- 172.21.15.107:0/2359466245 shutdown_connections 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2359466245 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8244102240 0x7f8244108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2359466245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244101880 0x7f8244101d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2359466245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244100690 0x7f8244100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 -- 172.21.15.107:0/2359466245 >> 172.21.15.107:0/2359466245 conn(0x7f82440fbc20 msgr2=0x7f82440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 -- 172.21.15.107:0/2359466245 shutdown_connections 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.636+0000 7f824a6bd640 1 -- 172.21.15.107:0/2359466245 wait complete. 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 Processor -- start 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 -- start start 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8244101880 0x7f824419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244102240 0x7f824419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f824410c370 con 0x7f8244100690 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f824410c1f0 con 0x7f8244102240 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f824a6bd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f824410c4f0 con 0x7f8244101880 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f8243fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f8243fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58792/0 (socket says 172.21.15.107:58792) 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f8243fff640 1 -- 172.21.15.107:0/2696111641 learned_addr learned my addr 172.21.15.107:0/2696111641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.637+0000 7f8248c5a640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244102240 0x7f824419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 -- 172.21.15.107:0/2696111641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8244101880 msgr2=0x7f824419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8244101880 0x7f824419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 -- 172.21.15.107:0/2696111641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244102240 msgr2=0x7f824419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244102240 0x7f824419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 -- 172.21.15.107:0/2696111641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824419f700 con 0x7f8244100690 2024-04-24T23:42:39.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f8243fff640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f823400c970 tx=0x7f823400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234007c10 con 0x7f8244100690 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8234007db0 con 0x7f8244100690 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824419f990 con 0x7f8244100690 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823400f940 con 0x7f8244100690 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.638+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f824419ff60 con 0x7f8244100690 2024-04-24T23:42:39.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.639+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8244107950 con 0x7f8244100690 2024-04-24T23:42:39.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.640+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f823400fae0 con 0x7f8244100690 2024-04-24T23:42:39.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.641+0000 7f82417fa640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 0x7f822c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:39.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.641+0000 7f82437fe640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 0x7f822c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:39.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.641+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f82340cf360 con 0x7f8244100690 2024-04-24T23:42:39.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.641+0000 7f82437fe640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 0x7f822c0a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82380096c0 tx=0x7f8238031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:39.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.645+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f823409c820 con 0x7f8244100690 2024-04-24T23:42:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.865+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f824402d170 con 0x7f8244100690 2024-04-24T23:42:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.866+0000 7f82417fa640 1 -- 172.21.15.107:0/2696111641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7f823409c120 con 0x7f8244100690 2024-04-24T23:42:39.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:39.865 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T23:42:39.866 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 11 2024-04-24T23:42:39.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 msgr2=0x7f822c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 0x7f822c0a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82380096c0 tx=0x7f8238031400 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 msgr2=0x7f824406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f823400c970 tx=0x7f823400ce30 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 shutdown_connections 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f822c0a2b90 0x7f822c0a5050 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8244102240 0x7f824419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8244101880 0x7f824419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 --2- 172.21.15.107:0/2696111641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8244100690 0x7f824406a910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 >> 172.21.15.107:0/2696111641 conn(0x7f82440fbc20 msgr2=0x7f8244104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 shutdown_connections 2024-04-24T23:42:39.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:39.868+0000 7f824a6bd640 1 -- 172.21.15.107:0/2696111641 wait complete. 2024-04-24T23:42:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:40 smithi107 ceph-mon[29118]: Deploying daemon mds.a on smithi107 2024-04-24T23:42:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:40 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6838/527084969,v1:172.21.15.107:6839/527084969] up:boot 2024-04-24T23:42:40.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:40 smithi107 ceph-mon[29118]: fsmap 10 up:standby 2024-04-24T23:42:40.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:40 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:42:40.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2696111641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:40 smithi195 ceph-mon[33019]: Deploying daemon mds.a on smithi107 2024-04-24T23:42:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:40 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6838/527084969,v1:172.21.15.107:6839/527084969] up:boot 2024-04-24T23:42:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:40 smithi195 ceph-mon[33019]: fsmap 10 up:standby 2024-04-24T23:42:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:40 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:42:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2696111641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:40 smithi170 ceph-mon[32714]: Deploying daemon mds.a on smithi107 2024-04-24T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:40 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6838/527084969,v1:172.21.15.107:6839/527084969] up:boot 2024-04-24T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:40 smithi170 ceph-mon[32714]: fsmap 10 up:standby 2024-04-24T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:40 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:42:40.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2696111641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:41.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:41 smithi107 ceph-mon[29118]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:41 smithi195 ceph-mon[33019]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:41 smithi170 ceph-mon[32714]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:41.627 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:42:41 smithi107 systemd[1]: Started Ceph mds.a for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:41.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 -- 172.21.15.107:0/3030146857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 msgr2=0x7f8724100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 --2- 172.21.15.107:0/3030146857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724100aa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8720009900 tx=0x7f872002a410 comp rx=0 tx=0).stop 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 -- 172.21.15.107:0/3030146857 shutdown_connections 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 --2- 172.21.15.107:0/3030146857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724102230 0x7f8724108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 --2- 172.21.15.107:0/3030146857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724101870 0x7f8724101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 --2- 172.21.15.107:0/3030146857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724100aa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 -- 172.21.15.107:0/3030146857 >> 172.21.15.107:0/3030146857 conn(0x7f87240fbbf0 msgr2=0x7f87240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 -- 172.21.15.107:0/3030146857 shutdown_connections 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.980+0000 7f872b2f7640 1 -- 172.21.15.107:0/3030146857 wait complete. 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 Processor -- start 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 -- start start 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724101870 0x7f8724196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724102230 0x7f872419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f872410c350 con 0x7f8724100680 2024-04-24T23:42:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f872410c1d0 con 0x7f8724101870 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f872b2f7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f872410c4d0 con 0x7f8724102230 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58820/0 (socket says 172.21.15.107:58820) 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 -- 172.21.15.107:0/197712642 learned_addr learned my addr 172.21.15.107:0/197712642 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8728892640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724101870 0x7f8724196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729894640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724102230 0x7f872419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 -- 172.21.15.107:0/197712642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724102230 msgr2=0x7f872419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724102230 0x7f872419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 -- 172.21.15.107:0/197712642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724101870 msgr2=0x7f8724196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724101870 0x7f8724196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 -- 172.21.15.107:0/197712642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872419e650 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.981+0000 7f8729093640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8720031610 tx=0x7f87200043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.982+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872003f030 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.982+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872419e8e0 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.982+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8720003ce0 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.982+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720035070 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.982+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872419ee50 con 0x7f8724100680 2024-04-24T23:42:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.983+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8724107a20 con 0x7f8724100680 2024-04-24T23:42:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.984+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8720042030 con 0x7f8724100680 2024-04-24T23:42:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.984+0000 7f871a7fc640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 0x7f87100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.985+0000 7f8728892640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 0x7f87100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.985+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f87200ef150 con 0x7f8724100680 2024-04-24T23:42:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.985+0000 7f8728892640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 0x7f87100a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8724197f40 tx=0x7f8714009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:41.989+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87200bc550 con 0x7f8724100680 2024-04-24T23:42:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.203+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f872402d170 con 0x7f8724100680 2024-04-24T23:42:42.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.204+0000 7f871a7fc640 1 -- 172.21.15.107:0/197712642 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9674 (secure 0 0 0) 0x7f87200bbe50 con 0x7f8724100680 2024-04-24T23:42:42.205 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:42.205 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T23:42:42.205 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2024-04-24T23:42:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 msgr2=0x7f87100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 0x7f87100a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8724197f40 tx=0x7f8714009110 comp rx=0 tx=0).stop 2024-04-24T23:42:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 msgr2=0x7f8724196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8720031610 tx=0x7f87200043b0 comp rx=0 tx=0).stop 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 shutdown_connections 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87100a2b90 0x7f87100a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8724102230 0x7f872419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8724101870 0x7f8724196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 --2- 172.21.15.107:0/197712642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8724100680 0x7f8724196a30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 >> 172.21.15.107:0/197712642 conn(0x7f87240fbbf0 msgr2=0x7f8724104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 shutdown_connections 2024-04-24T23:42:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:42.206+0000 7f872b2f7640 1 -- 172.21.15.107:0/197712642 wait complete. 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: Deploying daemon mds.b on smithi170 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:42.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6840/1612113576,v1:172.21.15.107:6841/1612113576] up:boot 2024-04-24T23:42:42.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: fsmap 11 up:standby 2024-04-24T23:42:42.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:42:42.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:42 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/197712642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:42.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: Deploying daemon mds.b on smithi170 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6840/1612113576,v1:172.21.15.107:6841/1612113576] up:boot 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: fsmap 11 up:standby 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:42:42.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:42 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/197712642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:42.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: Deploying daemon mds.b on smithi170 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6840/1612113576,v1:172.21.15.107:6841/1612113576] up:boot 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: fsmap 11 up:standby 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:42:42.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:42 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/197712642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:44.134 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 podman[63050]: 2024-04-24 23:42:44.135104625 +0000 UTC m=+0.152011293 container create 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True) 2024-04-24T23:42:44.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:44.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 -- 172.21.15.107:0/2144623045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac100680 msgr2=0x7fedac100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:44.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 --2- 172.21.15.107:0/2144623045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac100680 0x7fedac100aa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feda8009900 tx=0x7feda802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:44.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 -- 172.21.15.107:0/2144623045 shutdown_connections 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 --2- 172.21.15.107:0/2144623045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 0x7fedac108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 --2- 172.21.15.107:0/2144623045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac101870 0x7fedac101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 --2- 172.21.15.107:0/2144623045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac100680 0x7fedac100aa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 -- 172.21.15.107:0/2144623045 >> 172.21.15.107:0/2144623045 conn(0x7fedac0fbbf0 msgr2=0x7fedac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 -- 172.21.15.107:0/2144623045 shutdown_connections 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.329+0000 7fedb3377640 1 -- 172.21.15.107:0/2144623045 wait complete. 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 Processor -- start 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 -- start start 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 0x7fedac0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 0x7fedac19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10c350 con 0x7fedac101870 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10c1d0 con 0x7fedac100680 2024-04-24T23:42:44.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb3377640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10c4d0 con 0x7fedac102230 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58838/0 (socket says 172.21.15.107:58838) 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb1113640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 0x7fedac0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 -- 172.21.15.107:0/809220290 learned_addr learned my addr 172.21.15.107:0/809220290 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb1914640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 0x7fedac19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 -- 172.21.15.107:0/809220290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 msgr2=0x7fedac19edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 0x7fedac19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 -- 172.21.15.107:0/809220290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 msgr2=0x7fedac0694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 0x7fedac0694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 -- 172.21.15.107:0/809220290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedac19f690 con 0x7fedac101870 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb1113640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 0x7fedac0694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.330+0000 7fedb0912640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00d950 tx=0x7fed9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.331+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c017030 con 0x7fedac101870 2024-04-24T23:42:44.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.331+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedac19f980 con 0x7fedac101870 2024-04-24T23:42:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.331+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c004020 con 0x7fedac101870 2024-04-24T23:42:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.331+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0047a0 con 0x7fedac101870 2024-04-24T23:42:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.331+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedac19fed0 con 0x7fedac101870 2024-04-24T23:42:44.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.332+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedac1079a0 con 0x7fedac101870 2024-04-24T23:42:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.333+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fed9c0041d0 con 0x7fedac101870 2024-04-24T23:42:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.333+0000 7feda27fc640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 0x7fed980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.333+0000 7fedb1113640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 0x7fed980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.333+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fed9c0ce610 con 0x7fedac101870 2024-04-24T23:42:44.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.334+0000 7fedb1113640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 0x7fed980a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feda802aa00 tx=0x7feda80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.338+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed9c09ba10 con 0x7fedac101870 2024-04-24T23:42:44.405 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 podman[63050]: 2024-04-24 23:42:44.379643169 +0000 UTC m=+0.396549836 container init 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T23:42:44.405 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 podman[63050]: 2024-04-24 23:42:44.382480102 +0000 UTC m=+0.399386773 container start 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T23:42:44.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.555+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedac02d2e0 con 0x7fedac101870 2024-04-24T23:42:44.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.556+0000 7feda27fc640 1 -- 172.21.15.107:0/809220290 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9674 (secure 0 0 0) 0x7fed9c09b310 con 0x7fedac101870 2024-04-24T23:42:44.557 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:44.557 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T23:42:44.557 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 msgr2=0x7fed980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 0x7fed980a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feda802aa00 tx=0x7feda80315b0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 msgr2=0x7fedac19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00d950 tx=0x7fed9c00de10 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 shutdown_connections 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fed980a2b90 0x7fed980a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedac102230 0x7fedac19edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedac101870 0x7fedac19c760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 --2- 172.21.15.107:0/809220290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedac100680 0x7fedac0694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 >> 172.21.15.107:0/809220290 conn(0x7fedac0fbbf0 msgr2=0x7fedac104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 shutdown_connections 2024-04-24T23:42:44.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:44.558+0000 7fedb3377640 1 -- 172.21.15.107:0/809220290 wait complete. 2024-04-24T23:42:44.835 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 ceph-mds[63070]: -- 172.21.15.170:0/1085933335 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558624cb7a40 con 0x558624c8b000 2024-04-24T23:42:44.835 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: starting mds.b at 2024-04-24T23:42:44.835 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 bash[63050]: 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d 2024-04-24T23:42:44.835 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:44 smithi170 systemd[1]: Started Ceph mds.b for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/809220290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:45 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/809220290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:45 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:45.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:45.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/809220290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:45.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:45 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:46 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:boot 2024-04-24T23:42:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:46 smithi195 ceph-mon[33019]: fsmap 12 up:standby 2024-04-24T23:42:46.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:46 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:42:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:46 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:boot 2024-04-24T23:42:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:46 smithi170 ceph-mon[32714]: fsmap 12 up:standby 2024-04-24T23:42:46.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:46 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:42:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:46 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:boot 2024-04-24T23:42:46.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:46 smithi107 ceph-mon[29118]: fsmap 12 up:standby 2024-04-24T23:42:46.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:46 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:42:46.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 -- 172.21.15.107:0/571201408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 msgr2=0x7f9c6c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:46.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 --2- 172.21.15.107:0/571201408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c106f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009900 tx=0x7f9c5c02a410 comp rx=0 tx=0).stop 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 -- 172.21.15.107:0/571201408 shutdown_connections 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 --2- 172.21.15.107:0/571201408 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c107460 0x7f9c6c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 --2- 172.21.15.107:0/571201408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c106f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 --2- 172.21.15.107:0/571201408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c102200 0x7f9c6c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 -- 172.21.15.107:0/571201408 >> 172.21.15.107:0/571201408 conn(0x7f9c6c0fbbf0 msgr2=0x7f9c6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 -- 172.21.15.107:0/571201408 shutdown_connections 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.673+0000 7f9c73638640 1 -- 172.21.15.107:0/571201408 wait complete. 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 Processor -- start 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 -- start start 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c102200 0x7f9c6c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c107460 0x7f9c6c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:46.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c10d3f0 con 0x7f9c6c104b30 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c10d270 con 0x7f9c6c107460 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c73638640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c10d570 con 0x7f9c6c102200 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33494/0 (socket says 172.21.15.107:33494) 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 -- 172.21.15.107:0/2998885107 learned_addr learned my addr 172.21.15.107:0/2998885107 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c713d4640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c102200 0x7f9c6c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 -- 172.21.15.107:0/2998885107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c102200 msgr2=0x7f9c6c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c102200 0x7f9c6c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c71bd5640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c107460 0x7f9c6c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 -- 172.21.15.107:0/2998885107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c107460 msgr2=0x7f9c6c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c107460 0x7f9c6c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.674+0000 7f9c70bd3640 1 -- 172.21.15.107:0/2998885107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6c19f670 con 0x7f9c6c104b30 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c70bd3640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c031610 tx=0x7f9c5c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:46.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c03f030 con 0x7f9c6c104b30 2024-04-24T23:42:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c5c003ce0 con 0x7f9c6c104b30 2024-04-24T23:42:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c6c19f900 con 0x7f9c6c104b30 2024-04-24T23:42:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c035070 con 0x7f9c6c104b30 2024-04-24T23:42:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.675+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c6c19fdf0 con 0x7f9c6c104b30 2024-04-24T23:42:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.676+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c6c0fd6e0 con 0x7f9c6c104b30 2024-04-24T23:42:46.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.677+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9c5c042030 con 0x7f9c6c104b30 2024-04-24T23:42:46.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.677+0000 7f9c627fc640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 0x7f9c580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:46.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.677+0000 7f9c713d4640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 0x7f9c580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:46.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.677+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9c5c0ef250 con 0x7f9c6c104b30 2024-04-24T23:42:46.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.678+0000 7f9c713d4640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 0x7f9c580a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c68005c40 tx=0x7f9c6800a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:46.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.682+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9c5c0bc650 con 0x7f9c6c104b30 2024-04-24T23:42:46.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.895+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c6c02d3c0 con 0x7f9c6c104b30 2024-04-24T23:42:46.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.896+0000 7f9c627fc640 1 -- 172.21.15.107:0/2998885107 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10503 (secure 0 0 0) 0x7f9c5c0bbf50 con 0x7f9c6c104b30 2024-04-24T23:42:46.896 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:46.896 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-24T23:42:46.897 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 13 2024-04-24T23:42:46.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 msgr2=0x7f9c580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:46.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 0x7f9c580a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c68005c40 tx=0x7f9c6800a590 comp rx=0 tx=0).stop 2024-04-24T23:42:46.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 msgr2=0x7f9c6c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c031610 tx=0x7f9c5c0043b0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 shutdown_connections 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9c580a2b90 0x7f9c580a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6c107460 0x7f9c6c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c6c104b30 0x7f9c6c197fe0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 --2- 172.21.15.107:0/2998885107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9c6c102200 0x7f9c6c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.898+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 >> 172.21.15.107:0/2998885107 conn(0x7f9c6c0fbbf0 msgr2=0x7f9c6c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.899+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 shutdown_connections 2024-04-24T23:42:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:46.899+0000 7f9c73638640 1 -- 172.21.15.107:0/2998885107 wait complete. 2024-04-24T23:42:46.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:42:47.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 -- 172.21.15.107:0/362219445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 msgr2=0x7f8958108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 --2- 172.21.15.107:0/362219445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f8958108ac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8950009900 tx=0x7f895002a410 comp rx=0 tx=0).stop 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 -- 172.21.15.107:0/362219445 shutdown_connections 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 --2- 172.21.15.107:0/362219445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f8958108ac0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 --2- 172.21.15.107:0/362219445 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958101870 0x7f8958101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 --2- 172.21.15.107:0/362219445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958100680 0x7f8958100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.010+0000 7f895f770640 1 -- 172.21.15.107:0/362219445 >> 172.21.15.107:0/362219445 conn(0x7f89580fbbf0 msgr2=0x7f89580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.011+0000 7f895f770640 1 -- 172.21.15.107:0/362219445 shutdown_connections 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.011+0000 7f895f770640 1 -- 172.21.15.107:0/362219445 wait complete. 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.011+0000 7f895f770640 1 Processor -- start 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.011+0000 7f895f770640 1 -- start start 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.011+0000 7f895f770640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958100680 0x7f8958069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958101870 0x7f895819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f895810c2e0 con 0x7f8958102230 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f895810c160 con 0x7f8958101870 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f895810c460 con 0x7f8958100680 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33502/0 (socket says 172.21.15.107:33502) 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 -- 172.21.15.107:0/1859842403 learned_addr learned my addr 172.21.15.107:0/1859842403 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895cd0b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958101870 0x7f895819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895d50c640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958100680 0x7f8958069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 -- 172.21.15.107:0/1859842403 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958100680 msgr2=0x7f8958069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958100680 0x7f8958069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 -- 172.21.15.107:0/1859842403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958101870 msgr2=0x7f895819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958101870 0x7f895819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 -- 172.21.15.107:0/1859842403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895819f5d0 con 0x7f8958102230 2024-04-24T23:42:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895dd0d640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89500314e0 tx=0x7f8950031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895003f030 con 0x7f8958102230 2024-04-24T23:42:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8950003ad0 con 0x7f8958102230 2024-04-24T23:42:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895819f860 con 0x7f8958102230 2024-04-24T23:42:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950035070 con 0x7f8958102230 2024-04-24T23:42:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.012+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895819fd50 con 0x7f8958102230 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.014+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89581078d0 con 0x7f8958102230 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.014+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8950042030 con 0x7f8958102230 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.015+0000 7f894e7fc640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 0x7f89440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.015+0000 7f895d50c640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 0x7f89440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.015+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f89500ef180 con 0x7f8958102230 2024-04-24T23:42:47.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.015+0000 7f895d50c640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 0x7f89440a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89581016d0 tx=0x7f8954009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.019+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89500bc580 con 0x7f8958102230 2024-04-24T23:42:47.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.234+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8958062340 con 0x7f8958102230 2024-04-24T23:42:47.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.235+0000 7f894e7fc640 1 -- 172.21.15.107:0/1859842403 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7f89500bbe80 con 0x7f8958102230 2024-04-24T23:42:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:47.235 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:41:58.539760+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 msgr2=0x7f89440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 0x7f89440a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89581016d0 tx=0x7f8954009500 comp rx=0 tx=0).stop 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 msgr2=0x7f895819ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89500314e0 tx=0x7f8950031b60 comp rx=0 tx=0).stop 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 shutdown_connections 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89440a2b90 0x7f89440a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8958102230 0x7f895819ed10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8958101870 0x7f895819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 --2- 172.21.15.107:0/1859842403 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8958100680 0x7f8958069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 >> 172.21.15.107:0/1859842403 conn(0x7f89580fbbf0 msgr2=0x7f8958104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 shutdown_connections 2024-04-24T23:42:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.237+0000 7f895f770640 1 -- 172.21.15.107:0/1859842403 wait complete. 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2998885107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:47.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:47.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:47.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:47 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:42:47.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2998885107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:47.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:47.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:47.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:47 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2998885107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:47.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:47.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:47.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:47 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:47.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 -- 172.21.15.107:0/617346288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450102200 msgr2=0x7f04501045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 --2- 172.21.15.107:0/617346288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450102200 0x7f04501045f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0444009900 tx=0x7f044402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:47.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 -- 172.21.15.107:0/617346288 shutdown_connections 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 --2- 172.21.15.107:0/617346288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450107460 0x7f04501098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 --2- 172.21.15.107:0/617346288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 0x7f0450106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 --2- 172.21.15.107:0/617346288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450102200 0x7f04501045f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 -- 172.21.15.107:0/617346288 >> 172.21.15.107:0/617346288 conn(0x7f04500fbbf0 msgr2=0x7f04500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 -- 172.21.15.107:0/617346288 shutdown_connections 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.353+0000 7f0455a6d640 1 -- 172.21.15.107:0/617346288 wait complete. 2024-04-24T23:42:47.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 Processor -- start 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 -- start start 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450102200 0x7f0450197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 0x7f0450198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f045010d420 con 0x7f0450107460 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f045010d2a0 con 0x7f0450104b30 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f0455a6d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f045010d5a0 con 0x7f0450102200 2024-04-24T23:42:47.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f044ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f044ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33534/0 (socket says 172.21.15.107:33534) 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f044ffff640 1 -- 172.21.15.107:0/1040946269 learned_addr learned my addr 172.21.15.107:0/1040946269 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f044f7fe640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450102200 0x7f0450197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.354+0000 7f044effd640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 0x7f0450198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 -- 172.21.15.107:0/1040946269 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450102200 msgr2=0x7f0450197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450102200 0x7f0450197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 -- 172.21.15.107:0/1040946269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 msgr2=0x7f0450198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 0x7f0450198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 -- 172.21.15.107:0/1040946269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045019f710 con 0x7f0450107460 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044ffff640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f044800eb10 tx=0x7f044800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448004b20 con 0x7f0450107460 2024-04-24T23:42:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0448004cc0 con 0x7f0450107460 2024-04-24T23:42:47.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045019fa00 con 0x7f0450107460 2024-04-24T23:42:47.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448010760 con 0x7f0450107460 2024-04-24T23:42:47.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.355+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045019ff50 con 0x7f0450107460 2024-04-24T23:42:47.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.356+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04500fd6e0 con 0x7f0450107460 2024-04-24T23:42:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.358+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0448005000 con 0x7f0450107460 2024-04-24T23:42:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.358+0000 7f044cff9640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 0x7f04380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.359+0000 7f044f7fe640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 0x7f04380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.359+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f04480cf5c0 con 0x7f0450107460 2024-04-24T23:42:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.359+0000 7f044f7fe640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 0x7f04380a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04440314e0 tx=0x7f0444000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.361+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f044809c9c0 con 0x7f0450107460 2024-04-24T23:42:47.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.544+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f045002d3c0 con 0x7f0450107460 2024-04-24T23:42:47.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.545+0000 7f044cff9640 1 -- 172.21.15.107:0/1040946269 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f044809c2c0 con 0x7f0450107460 2024-04-24T23:42:47.545 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 msgr2=0x7f04380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 0x7f04380a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04440314e0 tx=0x7f0444000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 msgr2=0x7f045019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f044800eb10 tx=0x7f044800c2d0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 shutdown_connections 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f04380a2b90 0x7f04380a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0450107460 0x7f045019f0a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0450104b30 0x7f0450198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 --2- 172.21.15.107:0/1040946269 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0450102200 0x7f0450197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 >> 172.21.15.107:0/1040946269 conn(0x7f04500fbbf0 msgr2=0x7f0450102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 shutdown_connections 2024-04-24T23:42:47.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.547+0000 7f0455a6d640 1 -- 172.21.15.107:0/1040946269 wait complete. 2024-04-24T23:42:47.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T23:42:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 -- 172.21.15.107:0/2860333817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 msgr2=0x7f1a9c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2860333817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c101bd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009900 tx=0x7f1a8c02a410 comp rx=0 tx=0).stop 2024-04-24T23:42:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 -- 172.21.15.107:0/2860333817 shutdown_connections 2024-04-24T23:42:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2860333817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 0x7f1a9c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2860333817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 0x7f1a9c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2860333817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c101bd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 -- 172.21.15.107:0/2860333817 >> 172.21.15.107:0/2860333817 conn(0x7f1a9c0fcd20 msgr2=0x7f1a9c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 -- 172.21.15.107:0/2860333817 shutdown_connections 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.676+0000 7f1aa2058640 1 -- 172.21.15.107:0/2860333817 wait complete. 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 Processor -- start 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 -- start start 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 0x7f1a9c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 0x7f1a9c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10d430 con 0x7f1a9c1017b0 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10d2b0 con 0x7f1a9c1029a0 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1aa2058640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10d5b0 con 0x7f1a9c103360 2024-04-24T23:42:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33554/0 (socket says 172.21.15.107:33554) 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 -- 172.21.15.107:0/2770144510 learned_addr learned my addr 172.21.15.107:0/2770144510 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9affd640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 0x7f1a9c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9bfff640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 0x7f1a9c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 -- 172.21.15.107:0/2770144510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 msgr2=0x7f1a9c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 0x7f1a9c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 -- 172.21.15.107:0/2770144510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 msgr2=0x7f1a9c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 0x7f1a9c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.677+0000 7f1a9b7fe640 1 -- 172.21.15.107:0/2770144510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c19f700 con 0x7f1a9c1017b0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1a9b7fe640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c031610 tx=0x7f1a8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c03f030 con 0x7f1a9c1017b0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9c19f990 con 0x7f1a9c1017b0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a8c003ad0 con 0x7f1a9c1017b0 2024-04-24T23:42:47.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c035070 con 0x7f1a9c1017b0 2024-04-24T23:42:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.678+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9c19fed0 con 0x7f1a9c1017b0 2024-04-24T23:42:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.679+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9c108b50 con 0x7f1a9c1017b0 2024-04-24T23:42:47.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.680+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1a8c042030 con 0x7f1a9c1017b0 2024-04-24T23:42:47.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.680+0000 7f1a98ff9640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 0x7f1a840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:47.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.681+0000 7f1a9affd640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 0x7f1a840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:47.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.681+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1a8c0ef150 con 0x7f1a9c1017b0 2024-04-24T23:42:47.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.681+0000 7f1a9affd640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 0x7f1a840a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c198ff0 tx=0x7f1a90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:47.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.685+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a8c0bbee0 con 0x7f1a9c1017b0 2024-04-24T23:42:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.892+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1a9c062340 con 0x7f1a9c1017b0 2024-04-24T23:42:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.893+0000 7f1a98ff9640 1 -- 172.21.15.107:0/2770144510 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f1a8c0c01e0 con 0x7f1a9c1017b0 2024-04-24T23:42:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:42:47.893 INFO:teuthology.orchestra.run.smithi107.stdout:] 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.894+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 msgr2=0x7f1a840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.894+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 0x7f1a840a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c198ff0 tx=0x7f1a90005f90 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.894+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 msgr2=0x7f1a9c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.894+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c031610 tx=0x7f1a8c004270 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 shutdown_connections 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1a840a2b90 0x7f1a840a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1a9c103360 0x7f1a9c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a9c1029a0 0x7f1a9c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 --2- 172.21.15.107:0/2770144510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a9c1017b0 0x7f1a9c197ae0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 >> 172.21.15.107:0/2770144510 conn(0x7f1a9c0fcd20 msgr2=0x7f1a9c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 shutdown_connections 2024-04-24T23:42:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:47.895+0000 7f1aa2058640 1 -- 172.21.15.107:0/2770144510 wait complete. 2024-04-24T23:42:47.905 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T23:42:47.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 -- 172.21.15.107:0/1532845130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc1022a0 msgr2=0x7f64dc104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 --2- 172.21.15.107:0/1532845130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc1022a0 0x7f64dc104690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009900 tx=0x7f64d002a410 comp rx=0 tx=0).stop 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 -- 172.21.15.107:0/1532845130 shutdown_connections 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 --2- 172.21.15.107:0/1532845130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 0x7f64dc1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 --2- 172.21.15.107:0/1532845130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc104bd0 0x7f64dc106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 --2- 172.21.15.107:0/1532845130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc1022a0 0x7f64dc104690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 -- 172.21.15.107:0/1532845130 >> 172.21.15.107:0/1532845130 conn(0x7f64dc0fbc30 msgr2=0x7f64dc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:48.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 -- 172.21.15.107:0/1532845130 shutdown_connections 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 -- 172.21.15.107:0/1532845130 wait complete. 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.012+0000 7f64e3093640 1 Processor -- start 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 -- start start 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc1022a0 0x7f64dc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 0x7f64dc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d3f0 con 0x7f64dc104bd0 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d270 con 0x7f64dc107500 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d570 con 0x7f64dc1022a0 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33564/0 (socket says 172.21.15.107:33564) 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 -- 172.21.15.107:0/2232309977 learned_addr learned my addr 172.21.15.107:0/2232309977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:48.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e0e2f640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc1022a0 0x7f64dc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e1630640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 0x7f64dc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 -- 172.21.15.107:0/2232309977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc1022a0 msgr2=0x7f64dc197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc1022a0 0x7f64dc197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 -- 172.21.15.107:0/2232309977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 msgr2=0x7f64dc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 0x7f64dc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 -- 172.21.15.107:0/2232309977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc19f710 con 0x7f64dc104bd0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64dbfff640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9a0 tx=0x7f64c800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c800cd80 con 0x7f64dc104bd0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc19fa00 con 0x7f64dc104bd0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64c80035f0 con 0x7f64dc104bd0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.013+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8010690 con 0x7f64dc104bd0 2024-04-24T23:42:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.014+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc19ffa0 con 0x7f64dc104bd0 2024-04-24T23:42:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.015+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64dc0fd720 con 0x7f64dc104bd0 2024-04-24T23:42:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.016+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f64c80037a0 con 0x7f64dc104bd0 2024-04-24T23:42:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.017+0000 7f64d9ffb640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 0x7f64cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.017+0000 7f64e0e2f640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 0x7f64cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:48.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.017+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f64c80cee70 con 0x7f64dc104bd0 2024-04-24T23:42:48.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.017+0000 7f64e0e2f640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 0x7f64cc0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64d00314e0 tx=0x7f64d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:48.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.026+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f64c809c270 con 0x7f64dc104bd0 2024-04-24T23:42:48.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1859842403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1040946269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:48.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2770144510' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T23:42:48.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:48.192 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.192 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1859842403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.192 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1040946269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:48.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2770144510' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T23:42:48.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:48.197 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.197 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1859842403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:48.197 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1040946269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:42:48.197 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2770144510' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T23:42:48.197 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:48.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:48.211+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f64dc02d3c0 con 0x7f64dc104bd0 2024-04-24T23:42:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.045+0000 7f64d9ffb640 1 -- 172.21.15.107:0/2232309977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f64c809bb70 con 0x7f64dc104bd0 2024-04-24T23:42:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 msgr2=0x7f64cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 0x7f64cc0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64d00314e0 tx=0x7f64d0000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 msgr2=0x7f64dc198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9a0 tx=0x7f64c800eda0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 shutdown_connections 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64cc0a2b90 0x7f64cc0a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64dc107500 0x7f64dc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64dc104bd0 0x7f64dc198030 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 --2- 172.21.15.107:0/2232309977 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f64dc1022a0 0x7f64dc197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.047+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 >> 172.21.15.107:0/2232309977 conn(0x7f64dc0fbc30 msgr2=0x7f64dc102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.048+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 shutdown_connections 2024-04-24T23:42:49.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.048+0000 7f64e3093640 1 -- 172.21.15.107:0/2232309977 wait complete. 2024-04-24T23:42:49.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-24T23:42:49.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- 172.21.15.107:0/2589696909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c101890 msgr2=0x7fbb1c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:49.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/2589696909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c101890 0x7fbb1c101d10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009900 tx=0x7fbb0802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:49.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- 172.21.15.107:0/2589696909 shutdown_connections 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/2589696909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c102250 0x7fbb1c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/2589696909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c101890 0x7fbb1c101d10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/2589696909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c1006a0 0x7fbb1c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- 172.21.15.107:0/2589696909 >> 172.21.15.107:0/2589696909 conn(0x7fbb1c0fbc30 msgr2=0x7fbb1c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- 172.21.15.107:0/2589696909 shutdown_connections 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- 172.21.15.107:0/2589696909 wait complete. 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 Processor -- start 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- start start 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c1006a0 0x7fbb1c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 0x7fbb1c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c10d440 con 0x7fbb1c102250 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c10d2c0 con 0x7fbb1c1006a0 2024-04-24T23:42:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb22c9e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c10d5c0 con 0x7fbb1c101890 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb2123b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb2123b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33584/0 (socket says 172.21.15.107:33584) 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.166+0000 7fbb2123b640 1 -- 172.21.15.107:0/1624299555 learned_addr learned my addr 172.21.15.107:0/1624299555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb1bfff640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 0x7fbb1c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 -- 172.21.15.107:0/1624299555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 msgr2=0x7fbb1c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 0x7fbb1c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 -- 172.21.15.107:0/1624299555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c1006a0 msgr2=0x7fbb1c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb20a3a640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c1006a0 0x7fbb1c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c1006a0 0x7fbb1c197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 -- 172.21.15.107:0/1624299555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c19f6b0 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb1bfff640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 0x7fbb1c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb2123b640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400cce0 tx=0x7fbb1400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1400f450 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb1400fad0 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14018c10 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c19f9a0 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.167+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c19fea0 con 0x7fbb1c102250 2024-04-24T23:42:49.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.168+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c1079c0 con 0x7fbb1c102250 2024-04-24T23:42:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.169+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbb1400f5f0 con 0x7fbb1c102250 2024-04-24T23:42:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.170+0000 7fbb19ffb640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 0x7fbb0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.170+0000 7fbb20a3a640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 0x7fbb0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.170+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fbb140cf4a0 con 0x7fbb1c102250 2024-04-24T23:42:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.170+0000 7fbb20a3a640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 0x7fbb0c0a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbb100058c0 tx=0x7fbb10005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:49.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.174+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb14014030 con 0x7fbb1c102250 2024-04-24T23:42:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T23:42:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:49.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:49.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:49.351+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fbb1c02d170 con 0x7fbb1c102250 2024-04-24T23:42:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:42:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:42:50.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.051+0000 7fbb19ffb640 1 -- 172.21.15.107:0/1624299555 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fbb1409c710 con 0x7fbb1c102250 2024-04-24T23:42:50.051 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2024-04-24T23:42:50.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.053+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 msgr2=0x7fbb0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.053+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 0x7fbb0c0a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbb100058c0 tx=0x7fbb10005c80 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 msgr2=0x7fbb1c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400cce0 tx=0x7fbb1400a620 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 shutdown_connections 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb0c0a2b90 0x7fbb0c0a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb1c102250 0x7fbb1c19f040 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb1c101890 0x7fbb1c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 --2- 172.21.15.107:0/1624299555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb1c1006a0 0x7fbb1c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 >> 172.21.15.107:0/1624299555 conn(0x7fbb1c0fbc30 msgr2=0x7fbb1c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 shutdown_connections 2024-04-24T23:42:50.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.054+0000 7fbb22c9e640 1 -- 172.21.15.107:0/1624299555 wait complete. 2024-04-24T23:42:50.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T23:42:50.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 -- 172.21.15.107:0/4222630277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 msgr2=0x7ff000108b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 --2- 172.21.15.107:0/4222630277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000108b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feff8009cd0 tx=0x7feff802a8e0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 -- 172.21.15.107:0/4222630277 shutdown_connections 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 --2- 172.21.15.107:0/4222630277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000108b90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 --2- 172.21.15.107:0/4222630277 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000100930 0x7ff000100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 --2- 172.21.15.107:0/4222630277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000101c90 0x7ff0001020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 -- 172.21.15.107:0/4222630277 >> 172.21.15.107:0/4222630277 conn(0x7ff0000fb380 msgr2=0x7ff0000fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 -- 172.21.15.107:0/4222630277 shutdown_connections 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.174+0000 7ff0068a5640 1 -- 172.21.15.107:0/4222630277 wait complete. 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 Processor -- start 2024-04-24T23:42:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 -- start start 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000100930 0x7ff0001979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 0x7ff00019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff00010d470 con 0x7ff0001012f0 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff00010d2f0 con 0x7ff000100930 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7ff0068a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff00010d5f0 con 0x7ff000101c90 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7fefff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7fefff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33606/0 (socket says 172.21.15.107:33606) 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.175+0000 7fefff7fe640 1 -- 172.21.15.107:0/923059928 learned_addr learned my addr 172.21.15.107:0/923059928 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7feffffff640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000100930 0x7ff0001979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7ff004e42640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 0x7ff00019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 -- 172.21.15.107:0/923059928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 msgr2=0x7ff00019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 0x7ff00019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 -- 172.21.15.107:0/923059928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000100930 msgr2=0x7ff0001979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000100930 0x7ff0001979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 -- 172.21.15.107:0/923059928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00019f610 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7ff004e42640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 0x7ff00019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7fefff7fe640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feff000d770 tx=0x7feff000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0017070 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00019f900 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feff0005500 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0004e00 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.176+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00019fe00 con 0x7ff0001012f0 2024-04-24T23:42:50.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.177+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff000107a10 con 0x7ff0001012f0 2024-04-24T23:42:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.178+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7feff0004fa0 con 0x7ff0001012f0 2024-04-24T23:42:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.178+0000 7feffd7fa640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 0x7fefe80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.179+0000 7feffffff640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 0x7fefe80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.179+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7feff00ce9c0 con 0x7ff0001012f0 2024-04-24T23:42:50.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.179+0000 7feffffff640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 0x7fefe80a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feff40058c0 tx=0x7feff4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:50.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.183+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feff00d0e20 con 0x7ff0001012f0 2024-04-24T23:42:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:50 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T23:42:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:50 smithi195 ceph-mon[33019]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T23:42:50.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:50 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-24T23:42:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:50 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T23:42:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:50 smithi170 ceph-mon[32714]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T23:42:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:50 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-24T23:42:50.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:50 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2232309977' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T23:42:50.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:50 smithi107 ceph-mon[29118]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T23:42:50.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:50 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-24T23:42:50.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:50.403+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff0000623e0 con 0x7ff0001012f0 2024-04-24T23:42:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.280+0000 7feffd7fa640 1 -- 172.21.15.107:0/923059928 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7feff009bad0 con 0x7ff0001012f0 2024-04-24T23:42:51.281 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 msgr2=0x7fefe80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 0x7fefe80a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feff40058c0 tx=0x7feff4005850 comp rx=0 tx=0).stop 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 msgr2=0x7ff000197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feff000d770 tx=0x7feff000dc30 comp rx=0 tx=0).stop 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 shutdown_connections 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fefe80a2b90 0x7fefe80a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff000101c90 0x7ff00019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0001012f0 0x7ff000197f30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 --2- 172.21.15.107:0/923059928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff000100930 0x7ff0001979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 >> 172.21.15.107:0/923059928 conn(0x7ff0000fb380 msgr2=0x7ff000104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:51.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 shutdown_connections 2024-04-24T23:42:51.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.283+0000 7ff0068a5640 1 -- 172.21.15.107:0/923059928 wait complete. 2024-04-24T23:42:51.299 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-24T23:42:51.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-24T23:42:51.311 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: pgmap v283: 33 pgs: 15 creating+peering, 17 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-24T23:42:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T23:42:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T23:42:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:51 smithi195 ceph-mon[33019]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: pgmap v283: 33 pgs: 15 creating+peering, 17 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T23:42:51.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:51 smithi107 ceph-mon[29118]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T23:42:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: pgmap v283: 33 pgs: 15 creating+peering, 17 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1624299555' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-24T23:42:51.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T23:42:51.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T23:42:51.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:51 smithi170 ceph-mon[32714]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 -- 172.21.15.107:0/2547264111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9764102250 msgr2=0x7f9764108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2547264111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9764102250 0x7f9764108ae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f975c009ca0 tx=0x7f975c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 -- 172.21.15.107:0/2547264111 shutdown_connections 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2547264111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9764102250 0x7f9764108ae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2547264111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2547264111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f97641006a0 0x7f9764100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 -- 172.21.15.107:0/2547264111 >> 172.21.15.107:0/2547264111 conn(0x7f97640fbc30 msgr2=0x7f97640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 -- 172.21.15.107:0/2547264111 shutdown_connections 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 -- 172.21.15.107:0/2547264111 wait complete. 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.407+0000 7f976b7ce640 1 Processor -- start 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- start start 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 0x7f976419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f976410c3a0 con 0x7f97641006a0 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f976410c220 con 0x7f9764101890 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f976410c520 con 0x7f9764102250 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33616/0 (socket says 172.21.15.107:33616) 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 -- 172.21.15.107:0/2139626740 learned_addr learned my addr 172.21.15.107:0/2139626740 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f9768d69640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 -- 172.21.15.107:0/2139626740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 msgr2=0x7f976419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f9769d6b640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 0x7f976419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 0x7f976419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 -- 172.21.15.107:0/2139626740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 msgr2=0x7f9764198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 -- 172.21.15.107:0/2139626740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976419f740 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f9769d6b640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 0x7f976419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f9768d69640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976956a640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f975400d950 tx=0x7f975400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754017070 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9754009c30 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976419fa30 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975400f600 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.408+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976419ffb0 con 0x7f97641006a0 2024-04-24T23:42:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.409+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97641a01e0 con 0x7f97641006a0 2024-04-24T23:42:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.410+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f975401a070 con 0x7f97641006a0 2024-04-24T23:42:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.411+0000 7f975a7fc640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 0x7f97500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.411+0000 7f9768d69640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 0x7f97500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.411+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f97540cea30 con 0x7f97641006a0 2024-04-24T23:42:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.412+0000 7f9768d69640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 0x7f97500a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9764199030 tx=0x7f9760008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.416+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9754009de0 con 0x7f97641006a0 2024-04-24T23:42:51.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:51.600+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f97641a0910 con 0x7f97641006a0 2024-04-24T23:42:52.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.054+0000 7f975a7fc640 1 -- 172.21.15.107:0/2139626740 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f975409bae0 con 0x7f97641006a0 2024-04-24T23:42:52.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 msgr2=0x7f97500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:52.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 0x7f97500a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9764199030 tx=0x7f9760008040 comp rx=0 tx=0).stop 2024-04-24T23:42:52.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 msgr2=0x7f9764197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:52.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f975400d950 tx=0x7f975400de10 comp rx=0 tx=0).stop 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 shutdown_connections 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f97500a2b90 0x7f97500a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9764102250 0x7f976419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9764101890 0x7f9764198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 --2- 172.21.15.107:0/2139626740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97641006a0 0x7f9764197b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.056+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 >> 172.21.15.107:0/2139626740 conn(0x7f97640fbc30 msgr2=0x7f9764069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.057+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 shutdown_connections 2024-04-24T23:42:52.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.057+0000 7f976b7ce640 1 -- 172.21.15.107:0/2139626740 wait complete. 2024-04-24T23:42:52.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T23:42:52.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- 172.21.15.107:0/1082035670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c103340 msgr2=0x7ff66c109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:52.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 --2- 172.21.15.107:0/1082035670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c103340 0x7ff66c109bd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff664009ca0 tx=0x7ff66402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- 172.21.15.107:0/1082035670 shutdown_connections 2024-04-24T23:42:52.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 --2- 172.21.15.107:0/1082035670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c103340 0x7ff66c109bd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 --2- 172.21.15.107:0/1082035670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c102980 0x7ff66c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 --2- 172.21.15.107:0/1082035670 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c101790 0x7ff66c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- 172.21.15.107:0/1082035670 >> 172.21.15.107:0/1082035670 conn(0x7ff66c0fcd20 msgr2=0x7ff66c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- 172.21.15.107:0/1082035670 shutdown_connections 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- 172.21.15.107:0/1082035670 wait complete. 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 Processor -- start 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 -- start start 2024-04-24T23:42:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.216+0000 7ff6725d5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c101790 0x7ff66c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c103340 0x7ff66c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10d490 con 0x7ff66c102980 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10d310 con 0x7ff66c101790 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10d610 con 0x7ff66c103340 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33638/0 (socket says 172.21.15.107:33638) 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 -- 172.21.15.107:0/2649993673 learned_addr learned my addr 172.21.15.107:0/2649993673 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:52.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66bfff640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c101790 0x7ff66c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff670b72640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c103340 0x7ff66c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 -- 172.21.15.107:0/2649993673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c103340 msgr2=0x7ff66c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c103340 0x7ff66c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 -- 172.21.15.107:0/2649993673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c101790 msgr2=0x7ff66c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c101790 0x7ff66c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 -- 172.21.15.107:0/2649993673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c19f6c0 con 0x7ff66c102980 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff66b7fe640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b4a0 tx=0x7ff66000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:52.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660014070 con 0x7ff66c102980 2024-04-24T23:42:52.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c19f9b0 con 0x7ff66c102980 2024-04-24T23:42:52.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff660010af0 con 0x7ff66c102980 2024-04-24T23:42:52.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660017d10 con 0x7ff66c102980 2024-04-24T23:42:52.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.217+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c19ff80 con 0x7ff66c102980 2024-04-24T23:42:52.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.218+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff66c108ab0 con 0x7ff66c102980 2024-04-24T23:42:52.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.219+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff660017070 con 0x7ff66c102980 2024-04-24T23:42:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.220+0000 7ff6697fa640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 0x7ff6540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.220+0000 7ff66bfff640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 0x7ff6540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.220+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7ff6600ce720 con 0x7ff66c102980 2024-04-24T23:42:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.220+0000 7ff66bfff640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 0x7ff6540a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff65c005fd0 tx=0x7ff65c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.223+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff66009b000 con 0x7ff66c102980 2024-04-24T23:42:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:52 smithi195 ceph-mon[33019]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T23:42:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:52 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T23:42:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:52 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T23:42:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:52 smithi195 ceph-mon[33019]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T23:42:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:52 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-24T23:42:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:52 smithi170 ceph-mon[32714]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T23:42:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:52 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T23:42:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:52 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T23:42:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:52 smithi170 ceph-mon[32714]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T23:42:52.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:52 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-24T23:42:52.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:52 smithi107 ceph-mon[29118]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T23:42:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:52 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/923059928' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T23:42:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:52 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T23:42:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:52 smithi107 ceph-mon[29118]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T23:42:52.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:52 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-24T23:42:52.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:52.404+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff66c02d2c0 con 0x7ff66c102980 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: pgmap v286: 97 pgs: 19 creating+peering, 62 unknown, 16 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: Cluster is now healthy 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:active 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.313 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:53 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T23:42:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: pgmap v286: 97 pgs: 19 creating+peering, 62 unknown, 16 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-24T23:42:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: Cluster is now healthy 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:active 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:53 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T23:42:53.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: pgmap v286: 97 pgs: 19 creating+peering, 62 unknown, 16 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2139626740' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: Cluster is now healthy 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] up:active 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:53.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:53 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T23:42:54.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.296+0000 7ff6697fa640 1 -- 172.21.15.107:0/2649993673 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7ff66009f300 con 0x7ff66c102980 2024-04-24T23:42:54.297 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data_ec' created 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 msgr2=0x7ff6540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 0x7ff6540a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff65c005fd0 tx=0x7ff65c00b390 comp rx=0 tx=0).stop 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 msgr2=0x7ff66c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b4a0 tx=0x7ff66000b960 comp rx=0 tx=0).stop 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 shutdown_connections 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff6540a2b90 0x7ff6540a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff66c103340 0x7ff66c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66c102980 0x7ff66c197fe0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 --2- 172.21.15.107:0/2649993673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff66c101790 0x7ff66c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 >> 172.21.15.107:0/2649993673 conn(0x7ff66c0fcd20 msgr2=0x7ff66c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:54.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 shutdown_connections 2024-04-24T23:42:54.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.300+0000 7ff6725d5640 1 -- 172.21.15.107:0/2649993673 wait complete. 2024-04-24T23:42:54.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 -- 172.21.15.107:0/137922520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 msgr2=0x7fbb60109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 --2- 172.21.15.107:0/137922520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60109bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009ca0 tx=0x7fbb5802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 -- 172.21.15.107:0/137922520 shutdown_connections 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 --2- 172.21.15.107:0/137922520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60109bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 --2- 172.21.15.107:0/137922520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 0x7fbb60102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 --2- 172.21.15.107:0/137922520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 0x7fbb60101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 -- 172.21.15.107:0/137922520 >> 172.21.15.107:0/137922520 conn(0x7fbb600fcd20 msgr2=0x7fbb600ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 -- 172.21.15.107:0/137922520 shutdown_connections 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 -- 172.21.15.107:0/137922520 wait complete. 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.425+0000 7fbb65fee640 1 Processor -- start 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 -- start start 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 0x7fbb600754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 0x7fbb60073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:54.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6010d490 con 0x7fbb60103340 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6010d310 con 0x7fbb60102980 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb65fee640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6010d610 con 0x7fbb60101790 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33660/0 (socket says 172.21.15.107:33660) 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 -- 172.21.15.107:0/1254294535 learned_addr learned my addr 172.21.15.107:0/1254294535 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5f7fe640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 0x7fbb600754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5effd640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 0x7fbb60073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 -- 172.21.15.107:0/1254294535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 msgr2=0x7fbb600754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 0x7fbb600754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 -- 172.21.15.107:0/1254294535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 msgr2=0x7fbb60073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 0x7fbb60073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 -- 172.21.15.107:0/1254294535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb60074900 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5ffff640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb58005e30 tx=0x7fbb58031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.426+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58003c30 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.427+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb601a1c20 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.427+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb58003dd0 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.427+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5802fde0 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.427+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb601a2170 con 0x7fbb60103340 2024-04-24T23:42:54.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.428+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb60108ab0 con 0x7fbb60103340 2024-04-24T23:42:54.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.429+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbb58004050 con 0x7fbb60103340 2024-04-24T23:42:54.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.429+0000 7fbb5cff9640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 0x7fbb480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:54.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.430+0000 7fbb5f7fe640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 0x7fbb480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:54.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.430+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fbb580f0870 con 0x7fbb60103340 2024-04-24T23:42:54.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.430+0000 7fbb5f7fe640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 0x7fbb480a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:54.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.433+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb580bd550 con 0x7fbb60103340 2024-04-24T23:42:54.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:54 smithi195 ceph-mon[33019]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T23:42:54.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:54 smithi195 ceph-mon[33019]: pgmap v289: 97 pgs: 19 creating+peering, 27 unknown, 51 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-24T23:42:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:54 smithi170 ceph-mon[32714]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T23:42:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:54 smithi170 ceph-mon[32714]: pgmap v289: 97 pgs: 19 creating+peering, 27 unknown, 51 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-24T23:42:54.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:54.612+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fbb6002d170 con 0x7fbb60103340 2024-04-24T23:42:54.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:54 smithi107 ceph-mon[29118]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T23:42:54.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:54 smithi107 ceph-mon[29118]: pgmap v289: 97 pgs: 19 creating+peering, 27 unknown, 51 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-24T23:42:55.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.298+0000 7fbb5cff9640 1 -- 172.21.15.107:0/1254294535 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fbb580bce50 con 0x7fbb60103340 2024-04-24T23:42:55.298 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 4 allow_ec_overwrites to true 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.301+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 msgr2=0x7fbb480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.301+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 0x7fbb480a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb5400b390 comp rx=0 tx=0).stop 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.301+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 msgr2=0x7fbb60074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.301+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb58005e30 tx=0x7fbb58031ee0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 shutdown_connections 2024-04-24T23:42:55.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbb480a2b90 0x7fbb480a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb60103340 0x7fbb60074070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb60102980 0x7fbb60073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 --2- 172.21.15.107:0/1254294535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbb60101790 0x7fbb600754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 >> 172.21.15.107:0/1254294535 conn(0x7fbb600fcd20 msgr2=0x7fbb60105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 shutdown_connections 2024-04-24T23:42:55.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.302+0000 7fbb65fee640 1 -- 172.21.15.107:0/1254294535 wait complete. 2024-04-24T23:42:55.309 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:42:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: 2024-04-24T23:42:55.030+0000 7f0f86c79700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:42:55.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.430+0000 7f22d0363640 1 -- 172.21.15.107:0/729156301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 msgr2=0x7f22c8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.430+0000 7f22d0363640 1 --2- 172.21.15.107:0/729156301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c8108ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009ca0 tx=0x7f22c402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- 172.21.15.107:0/729156301 shutdown_connections 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- 172.21.15.107:0/729156301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c8108ac0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- 172.21.15.107:0/729156301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- 172.21.15.107:0/729156301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 0x7f22c8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- 172.21.15.107:0/729156301 >> 172.21.15.107:0/729156301 conn(0x7f22c80fbbf0 msgr2=0x7f22c80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- 172.21.15.107:0/729156301 shutdown_connections 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- 172.21.15.107:0/729156301 wait complete. 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 Processor -- start 2024-04-24T23:42:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- start start 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 0x7f22c8069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f22c810c300 con 0x7f22c8102230 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f22c810c180 con 0x7f22c8101870 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22d0363640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f22c810c480 con 0x7f22c8100680 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22ce900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22ce900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33672/0 (socket says 172.21.15.107:33672) 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.431+0000 7f22ce900640 1 -- 172.21.15.107:0/4181570996 learned_addr learned my addr 172.21.15.107:0/4181570996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce0ff640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 0x7f22c8069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22cd8fe640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 -- 172.21.15.107:0/4181570996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 msgr2=0x7f22c8069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 0x7f22c8069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 -- 172.21.15.107:0/4181570996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 msgr2=0x7f22c819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 -- 172.21.15.107:0/4181570996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c819f6f0 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22cd8fe640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22ce900640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005e30 tx=0x7f22c4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4003c30 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c819f920 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c4003dd0 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c402fde0 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.432+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c819fe70 con 0x7f22c8102230 2024-04-24T23:42:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.433+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c8107a20 con 0x7f22c8102230 2024-04-24T23:42:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.434+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f22c4004050 con 0x7f22c8102230 2024-04-24T23:42:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.435+0000 7f22bf7fe640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 0x7f22b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.435+0000 7f22ce0ff640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 0x7f22b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.435+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f22c40f07b0 con 0x7f22c8102230 2024-04-24T23:42:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.435+0000 7f22ce0ff640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 0x7f22b40a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f22c0005fd0 tx=0x7f22c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:55.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.438+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f22c40bd4f0 con 0x7f22c8102230 2024-04-24T23:42:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:55 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T23:42:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:55 smithi195 ceph-mon[33019]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T23:42:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:55 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-24T23:42:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:55 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T23:42:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:55 smithi170 ceph-mon[32714]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T23:42:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:55 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-24T23:42:55.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:55 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2649993673' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T23:42:55.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:55 smithi107 ceph-mon[29118]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T23:42:55.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:55 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-24T23:42:55.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:55.654+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f22c802d170 con 0x7f22c8102230 2024-04-24T23:42:56.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.319+0000 7f22bf7fe640 1 -- 172.21.15.107:0/4181570996 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f22c40bcdf0 con 0x7f22c8102230 2024-04-24T23:42:56.319 INFO:teuthology.orchestra.run.smithi107.stderr:added data pool 4 to fsmap 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 msgr2=0x7f22b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 0x7f22b40a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f22c0005fd0 tx=0x7f22c000b390 comp rx=0 tx=0).stop 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 msgr2=0x7f22c819ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005e30 tx=0x7f22c4031ee0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 shutdown_connections 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f22b40a2b90 0x7f22b40a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22c8102230 0x7f22c819ede0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22c8101870 0x7f22c819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 --2- 172.21.15.107:0/4181570996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f22c8100680 0x7f22c8069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 >> 172.21.15.107:0/4181570996 conn(0x7f22c80fbbf0 msgr2=0x7f22c8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:56.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 shutdown_connections 2024-04-24T23:42:56.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.321+0000 7f22d0363640 1 -- 172.21.15.107:0/4181570996 wait complete. 2024-04-24T23:42:56.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- 172.21.15.107:0/1812365360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b40100920 msgr2=0x7f0b40100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 --2- 172.21.15.107:0/1812365360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b40100920 0x7f0b40100da0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009930 tx=0x7f0b3402a420 comp rx=0 tx=0).stop 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- 172.21.15.107:0/1812365360 shutdown_connections 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 --2- 172.21.15.107:0/1812365360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b401012e0 0x7f0b40108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 --2- 172.21.15.107:0/1812365360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b40100920 0x7f0b40100da0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 --2- 172.21.15.107:0/1812365360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40101c80 0x7f0b401020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- 172.21.15.107:0/1812365360 >> 172.21.15.107:0/1812365360 conn(0x7f0b400fb3b0 msgr2=0x7f0b400fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:56.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- 172.21.15.107:0/1812365360 shutdown_connections 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- 172.21.15.107:0/1812365360 wait complete. 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 Processor -- start 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.434+0000 7f0b467a2640 1 -- start start 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40100920 0x7f0b401003a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b40101c80 0x7f0b4019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4010c340 con 0x7f0b401012e0 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4010c1c0 con 0x7f0b40100920 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b467a2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4010c4c0 con 0x7f0b40101c80 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33690/0 (socket says 172.21.15.107:33690) 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 -- 172.21.15.107:0/572748366 learned_addr learned my addr 172.21.15.107:0/572748366 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:56.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3ffff640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40100920 0x7f0b401003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b44d3f640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b40101c80 0x7f0b4019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 -- 172.21.15.107:0/572748366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b40101c80 msgr2=0x7f0b4019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b40101c80 0x7f0b4019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 -- 172.21.15.107:0/572748366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40100920 msgr2=0x7f0b401003a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40100920 0x7f0b401003a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 -- 172.21.15.107:0/572748366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4019f6e0 con 0x7f0b401012e0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3f7fe640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0b34031610 tx=0x7f0b340043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.435+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3403f030 con 0x7f0b401012e0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.436+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4019f970 con 0x7f0b401012e0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.436+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b34003ce0 con 0x7f0b401012e0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.436+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34035070 con 0x7f0b401012e0 2024-04-24T23:42:56.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.436+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4019feb0 con 0x7f0b401012e0 2024-04-24T23:42:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.437+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b40107a60 con 0x7f0b401012e0 2024-04-24T23:42:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.438+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0b34042030 con 0x7f0b401012e0 2024-04-24T23:42:56.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.438+0000 7f0b3d7fa640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 0x7f0b280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.438+0000 7f0b3ffff640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 0x7f0b280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.438+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0b340ef860 con 0x7f0b401012e0 2024-04-24T23:42:56.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.439+0000 7f0b3ffff640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 0x7f0b280a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b40069b60 tx=0x7f0b3000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:56.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.443+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b340bbf30 con 0x7f0b401012e0 2024-04-24T23:42:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:56 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-24T23:42:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:56 smithi170 ceph-mon[32714]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T23:42:56.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:56 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-24T23:42:56.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:56 smithi170 ceph-mon[32714]: pgmap v292: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-24T23:42:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:56 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-24T23:42:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:56 smithi107 ceph-mon[29118]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T23:42:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:56 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-24T23:42:56.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:56 smithi107 ceph-mon[29118]: pgmap v292: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-24T23:42:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.669+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b4002d170 con 0x7f0b401012e0 2024-04-24T23:42:56.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.670+0000 7f0b3d7fa640 1 -- 172.21.15.107:0/572748366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11821 (secure 0 0 0) 0x7f0b340c0230 con 0x7f0b401012e0 2024-04-24T23:42:56.670 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:56.670 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:42:55.655699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24506,"qdb_cluster":[24506]},"id":1}]} 2024-04-24T23:42:56.670 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 msgr2=0x7f0b280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 0x7f0b280a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b40069b60 tx=0x7f0b3000b390 comp rx=0 tx=0).stop 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 msgr2=0x7f0b4019c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0b34031610 tx=0x7f0b340043b0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 shutdown_connections 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b280a2b90 0x7f0b280a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b40101c80 0x7f0b4019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b401012e0 0x7f0b4019c7e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 --2- 172.21.15.107:0/572748366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b40100920 0x7f0b401003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 >> 172.21.15.107:0/572748366 conn(0x7f0b400fb3b0 msgr2=0x7f0b40104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:56.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 shutdown_connections 2024-04-24T23:42:56.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.672+0000 7f0b467a2640 1 -- 172.21.15.107:0/572748366 wait complete. 2024-04-24T23:42:56.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 -- 172.21.15.107:0/144916420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0100680 msgr2=0x7f84e0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 --2- 172.21.15.107:0/144916420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0100680 0x7f84e0100aa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009900 tx=0x7f84d802a410 comp rx=0 tx=0).stop 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 -- 172.21.15.107:0/144916420 shutdown_connections 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 --2- 172.21.15.107:0/144916420 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 0x7f84e0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 --2- 172.21.15.107:0/144916420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0101870 0x7f84e0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 --2- 172.21.15.107:0/144916420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0100680 0x7f84e0100aa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.780+0000 7f84e4aba640 1 -- 172.21.15.107:0/144916420 >> 172.21.15.107:0/144916420 conn(0x7f84e00fbbf0 msgr2=0x7f84e00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- 172.21.15.107:0/144916420 shutdown_connections 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- 172.21.15.107:0/144916420 wait complete. 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 Processor -- start 2024-04-24T23:42:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- start start 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 0x7f84e0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 0x7f84e019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84e010c320 con 0x7f84e0101870 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f84e010c1a0 con 0x7f84e0100680 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84e4aba640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f84e010c4a0 con 0x7f84e0102230 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.781+0000 7f84ddd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35370/0 (socket says 172.21.15.107:35370) 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 -- 172.21.15.107:0/1746092209 learned_addr learned my addr 172.21.15.107:0/1746092209 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84de59c640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 0x7f84e0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ded9d640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 0x7f84e019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 -- 172.21.15.107:0/1746092209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 msgr2=0x7f84e019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 0x7f84e019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 -- 172.21.15.107:0/1746092209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 msgr2=0x7f84e0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 0x7f84e0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 -- 172.21.15.107:0/1746092209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e019f6e0 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84de59c640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 0x7f84e0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84ddd9b640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84d400ebc0 tx=0x7f84d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4010630 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e019f9d0 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d4010cb0 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4018bd0 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.782+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e019ff20 con 0x7f84e0101870 2024-04-24T23:42:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.783+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e01a0150 con 0x7f84e0101870 2024-04-24T23:42:56.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.784+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f84d40107d0 con 0x7f84e0101870 2024-04-24T23:42:56.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.785+0000 7f84cf7fe640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 0x7f84c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:56.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.785+0000 7f84de59c640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 0x7f84c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:56.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.785+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84d40cfa00 con 0x7f84e0101870 2024-04-24T23:42:56.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.785+0000 7f84de59c640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 0x7f84c80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84d802aa00 tx=0x7f84d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.789+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84d409c720 con 0x7f84e0101870 2024-04-24T23:42:56.811 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:56 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1254294535' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-24T23:42:56.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:56 smithi195 ceph-mon[33019]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T23:42:56.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:56 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-24T23:42:56.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:56 smithi195 ceph-mon[33019]: pgmap v292: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-24T23:42:56.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.965+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84e01a0840 con 0x7f84e0101870 2024-04-24T23:42:56.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.967+0000 7f84cf7fe640 1 -- 172.21.15.107:0/1746092209 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f84d40bc0e0 con 0x7f84e0101870 2024-04-24T23:42:56.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:56.967 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 msgr2=0x7f84c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 0x7f84c80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f84d802aa00 tx=0x7f84d8000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 msgr2=0x7f84e0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84d400ebc0 tx=0x7f84d400c6a0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 shutdown_connections 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84c80a2b90 0x7f84c80a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f84e0102230 0x7f84e019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84e0101870 0x7f84e0198000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 --2- 172.21.15.107:0/1746092209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84e0100680 0x7f84e0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 >> 172.21.15.107:0/1746092209 conn(0x7f84e00fbbf0 msgr2=0x7f84e00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 shutdown_connections 2024-04-24T23:42:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:56.969+0000 7f84e4aba640 1 -- 172.21.15.107:0/1746092209 wait complete. 2024-04-24T23:42:56.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 -- 172.21.15.107:0/358801175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 msgr2=0x7f44441004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 --2- 172.21.15.107:0/358801175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f44441004b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4434009900 tx=0x7f443402a410 comp rx=0 tx=0).stop 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 -- 172.21.15.107:0/358801175 shutdown_connections 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 --2- 172.21.15.107:0/358801175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4444101390 0x7f4444101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 --2- 172.21.15.107:0/358801175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44441009f0 0x7f4444100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 --2- 172.21.15.107:0/358801175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f44441004b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 -- 172.21.15.107:0/358801175 >> 172.21.15.107:0/358801175 conn(0x7f44440fbbf0 msgr2=0x7f44440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 -- 172.21.15.107:0/358801175 shutdown_connections 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.079+0000 7f444a370640 1 -- 172.21.15.107:0/358801175 wait complete. 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 Processor -- start 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 -- start start 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f4444199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4444101390 0x7f44441a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f444410f5e0 con 0x7f44441009f0 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f444410f460 con 0x7f4444101390 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f444a370640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f444410f760 con 0x7f4444100090 2024-04-24T23:42:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f44437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f44437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35388/0 (socket says 172.21.15.107:35388) 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f44437fe640 1 -- 172.21.15.107:0/260176437 learned_addr learned my addr 172.21.15.107:0/260176437 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.080+0000 7f4443fff640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f4444199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f444890d640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4444101390 0x7f44441a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 -- 172.21.15.107:0/260176437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 msgr2=0x7f4444199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f4444199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 -- 172.21.15.107:0/260176437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4444101390 msgr2=0x7f44441a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4444101390 0x7f44441a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 -- 172.21.15.107:0/260176437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44441a18c0 con 0x7f44441009f0 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44437fe640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f442c00e9f0 tx=0x7f442c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c00cd80 con 0x7f44441009f0 2024-04-24T23:42:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44441a1bb0 con 0x7f44441009f0 2024-04-24T23:42:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442c005500 con 0x7f44441009f0 2024-04-24T23:42:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c010690 con 0x7f44441009f0 2024-04-24T23:42:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.081+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44441a20b0 con 0x7f44441009f0 2024-04-24T23:42:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.082+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444410b030 con 0x7f44441009f0 2024-04-24T23:42:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.083+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f442c0046c0 con 0x7f44441009f0 2024-04-24T23:42:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.084+0000 7f44417fa640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 0x7f44300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.084+0000 7f4443fff640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 0x7f44300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.084+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f442c0cf7c0 con 0x7f44441009f0 2024-04-24T23:42:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.084+0000 7f4443fff640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 0x7f44300a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4434031610 tx=0x7f4434000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.088+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f442c09c4e0 con 0x7f44441009f0 2024-04-24T23:42:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.264+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44440623e0 con 0x7f44441009f0 2024-04-24T23:42:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.266+0000 7f44417fa640 1 -- 172.21.15.107:0/260176437 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f442c09bde0 con 0x7f44441009f0 2024-04-24T23:42:57.266 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:57.266 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 msgr2=0x7f44300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 0x7f44300a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4434031610 tx=0x7f4434000bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 msgr2=0x7f444419a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f442c00e9f0 tx=0x7f442c00edf0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 shutdown_connections 2024-04-24T23:42:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f44300a2b90 0x7f44300a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4444101390 0x7f44441a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44441009f0 0x7f444419a1c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 --2- 172.21.15.107:0/260176437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4444100090 0x7f4444199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 >> 172.21.15.107:0/260176437 conn(0x7f44440fbbf0 msgr2=0x7f44440fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 shutdown_connections 2024-04-24T23:42:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.268+0000 7f444a370640 1 -- 172.21.15.107:0/260176437 wait complete. 2024-04-24T23:42:57.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:42:57.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 -- 172.21.15.107:0/3454300421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 msgr2=0x7f7fb80fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/3454300421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb80fff10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009930 tx=0x7f7fa802a420 comp rx=0 tx=0).stop 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 -- 172.21.15.107:0/3454300421 shutdown_connections 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/3454300421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8103480 0x7f7fb8109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/3454300421 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8102ac0 0x7f7fb8102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/3454300421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb80fff10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 -- 172.21.15.107:0/3454300421 >> 172.21.15.107:0/3454300421 conn(0x7f7fb80fb420 msgr2=0x7f7fb80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 -- 172.21.15.107:0/3454300421 shutdown_connections 2024-04-24T23:42:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.396+0000 7f7fbc91d640 1 -- 172.21.15.107:0/3454300421 wait complete. 2024-04-24T23:42:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 Processor -- start 2024-04-24T23:42:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 -- start start 2024-04-24T23:42:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8102ac0 0x7f7fb81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 0x7f7fb819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810d4c0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810d340 con 0x7f7fb8102ac0 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fbc91d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb810d640 con 0x7f7fb8103480 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35398/0 (socket says 172.21.15.107:35398) 2024-04-24T23:42:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 -- 172.21.15.107:0/2632224803 learned_addr learned my addr 172.21.15.107:0/2632224803 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb5d9b640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8102ac0 0x7f7fb81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb6d9d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 0x7f7fb819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 -- 172.21.15.107:0/2632224803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 msgr2=0x7f7fb819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 0x7f7fb819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 -- 172.21.15.107:0/2632224803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8102ac0 msgr2=0x7f7fb81980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8102ac0 0x7f7fb81980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 -- 172.21.15.107:0/2632224803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb819f7b0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb6d9d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 0x7f7fb819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:42:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fb659c640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8031440 tx=0x7f7fa8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.397+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa803f030 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.398+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa8003ad0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.398+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8035070 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.398+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb819fa40 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.398+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb819ff80 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.398+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb8108c90 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.399+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7fa8042030 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.400+0000 7f7fa77fe640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 0x7f7fa00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.400+0000 7f7fb5d9b640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 0x7f7fa00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.400+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7fa80ef8c0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.400+0000 7f7fb5d9b640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 0x7f7fa00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fb81990a0 tx=0x7f7fac00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.404+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7fa80bc5e0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T23:42:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-24T23:42:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/572748366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1746092209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:57 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/260176437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.616+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fb802d170 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.617+0000 7f7fa77fe640 1 -- 172.21.15.107:0/2632224803 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11821 (secure 0 0 0) 0x7f7fa80bbee0 con 0x7f7fb80ffaf0 2024-04-24T23:42:57.617 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:57.617 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:42:55.655699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24506,"qdb_cluster":[24506]},"id":1}]} 2024-04-24T23:42:57.618 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2024-04-24T23:42:57.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 msgr2=0x7f7fa00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 0x7f7fa00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fb81990a0 tx=0x7f7fac00b390 comp rx=0 tx=0).stop 2024-04-24T23:42:57.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 msgr2=0x7f7fb8197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8031440 tx=0x7f7fa8004270 comp rx=0 tx=0).stop 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 shutdown_connections 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7fa00a2c10 0x7f7fa00a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7fb8103480 0x7f7fb819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8102ac0 0x7f7fb81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 --2- 172.21.15.107:0/2632224803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fb80ffaf0 0x7f7fb8197b90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 >> 172.21.15.107:0/2632224803 conn(0x7f7fb80fb420 msgr2=0x7f7fb8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 shutdown_connections 2024-04-24T23:42:57.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.619+0000 7f7fbc91d640 1 -- 172.21.15.107:0/2632224803 wait complete. 2024-04-24T23:42:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T23:42:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-24T23:42:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/572748366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1746092209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:57 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/260176437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:42:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 -- 172.21.15.107:0/1128874758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 msgr2=0x7f3f24102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 --2- 172.21.15.107:0/1128874758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24102ea0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009930 tx=0x7f3f1802a420 comp rx=0 tx=0).stop 2024-04-24T23:42:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 -- 172.21.15.107:0/1128874758 shutdown_connections 2024-04-24T23:42:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 --2- 172.21.15.107:0/1128874758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f241033e0 0x7f3f24109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 --2- 172.21.15.107:0/1128874758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24102ea0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 --2- 172.21.15.107:0/1128874758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f240ffa50 0x7f3f240ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 -- 172.21.15.107:0/1128874758 >> 172.21.15.107:0/1128874758 conn(0x7f3f240fb380 msgr2=0x7f3f240fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 -- 172.21.15.107:0/1128874758 shutdown_connections 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.728+0000 7f3f28e88640 1 -- 172.21.15.107:0/1128874758 wait complete. 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 Processor -- start 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 -- start start 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f240ffa50 0x7f3f24197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f241033e0 0x7f3f2419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2410d430 con 0x7f3f24102a20 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2410d2b0 con 0x7f3f240ffa50 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f28e88640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2410d5b0 con 0x7f3f241033e0 2024-04-24T23:42:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35412/0 (socket says 172.21.15.107:35412) 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 -- 172.21.15.107:0/456302734 learned_addr learned my addr 172.21.15.107:0/456302734 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 -- 172.21.15.107:0/456302734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f241033e0 msgr2=0x7f3f2419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f241033e0 0x7f3f2419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 -- 172.21.15.107:0/456302734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f240ffa50 msgr2=0x7f3f24197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.729+0000 7f3f21d9b640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f240ffa50 0x7f3f24197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f21d9b640 1 -- 172.21.15.107:0/456302734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2419f700 con 0x7f3f24102a20 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f21d9b640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f18031610 tx=0x7f3f180043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1803f030 con 0x7f3f24102a20 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f18003ce0 con 0x7f3f24102a20 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2419f990 con 0x7f3f24102a20 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18035070 con 0x7f3f24102a20 2024-04-24T23:42:57.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.730+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2419fe80 con 0x7f3f24102a20 2024-04-24T23:42:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.731+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f24108c70 con 0x7f3f24102a20 2024-04-24T23:42:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.732+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3f18042030 con 0x7f3f24102a20 2024-04-24T23:42:57.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.732+0000 7f3f137fe640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 0x7f3f0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.732+0000 7f3f2259c640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 0x7f3f0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.732+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f180ef990 con 0x7f3f24102a20 2024-04-24T23:42:57.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.733+0000 7f3f2259c640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 0x7f3f0c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3f24069c40 tx=0x7f3f14005c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:57.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.737+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3f180bc6b0 con 0x7f3f24102a20 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4181570996' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/572748366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1746092209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:57 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/260176437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:57.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.917+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f24062340 con 0x7f3f24102a20 2024-04-24T23:42:57.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.918+0000 7f3f137fe640 1 -- 172.21.15.107:0/456302734 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f3f1809ae20 con 0x7f3f24102a20 2024-04-24T23:42:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:42:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:42:57.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 msgr2=0x7f3f0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 0x7f3f0c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3f24069c40 tx=0x7f3f14005c10 comp rx=0 tx=0).stop 2024-04-24T23:42:57.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 msgr2=0x7f3f24198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:57.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f18031610 tx=0x7f3f180043b0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 shutdown_connections 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f0c0a2b90 0x7f3f0c0a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f241033e0 0x7f3f2419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24102a20 0x7f3f24198020 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 --2- 172.21.15.107:0/456302734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f240ffa50 0x7f3f24197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.920+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 >> 172.21.15.107:0/456302734 conn(0x7f3f240fb380 msgr2=0x7f3f24105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.921+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 shutdown_connections 2024-04-24T23:42:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:57.921+0000 7f3f28e88640 1 -- 172.21.15.107:0/456302734 wait complete. 2024-04-24T23:42:57.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:42:50.405146+0000', 'modified': '2024-04-24T23:42:55.655699+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24506}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6839/2021576737', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 2021576737}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 2021576737}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24506, 'qdb_cluster': [24506]} 2024-04-24T23:42:57.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:42:57.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- 172.21.15.107:0/246697795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 msgr2=0x7f618c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- 172.21.15.107:0/246697795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c108240 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f617c009900 tx=0x7f617c02a410 comp rx=0 tx=0).stop 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- 172.21.15.107:0/246697795 shutdown_connections 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- 172.21.15.107:0/246697795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c108240 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- 172.21.15.107:0/246697795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f618c100f60 0x7f618c1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- 172.21.15.107:0/246697795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c0ffe10 0x7f618c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- 172.21.15.107:0/246697795 >> 172.21.15.107:0/246697795 conn(0x7f618c0fb380 msgr2=0x7f618c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- 172.21.15.107:0/246697795 shutdown_connections 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- 172.21.15.107:0/246697795 wait complete. 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 Processor -- start 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 -- start start 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f618c0ffe10 0x7f618c1097a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.031+0000 7f6190b48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c100f60 0x7f618c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c3c0 con 0x7f618c101920 2024-04-24T23:42:58.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c240 con 0x7f618c100f60 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c540 con 0x7f618c0ffe10 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35424/0 (socket says 172.21.15.107:35424) 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 -- 172.21.15.107:0/2405555209 learned_addr learned my addr 172.21.15.107:0/2405555209 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6189d9b640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c100f60 0x7f618c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 -- 172.21.15.107:0/2405555209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f618c0ffe10 msgr2=0x7f618c1097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f618c0ffe10 0x7f618c1097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 -- 172.21.15.107:0/2405555209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c100f60 msgr2=0x7f618c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c100f60 0x7f618c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 -- 172.21.15.107:0/2405555209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c19f750 con 0x7f618c101920 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f618ad9d640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f617c0314e0 tx=0x7f617c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f61837fe640 1 -- 172.21.15.107:0/2405555209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c03f030 con 0x7f618c101920 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f61837fe640 1 -- 172.21.15.107:0/2405555209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617c003ce0 con 0x7f618c101920 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c19f980 con 0x7f618c101920 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f61837fe640 1 -- 172.21.15.107:0/2405555209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c035070 con 0x7f618c101920 2024-04-24T23:42:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.032+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c19fef0 con 0x7f618c101920 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.034+0000 7f61837fe640 1 -- 172.21.15.107:0/2405555209 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f617c042030 con 0x7f618c101920 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 Processor -- start 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 -- start start 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d99e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9a3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56489d901f00 con 0x56489d9a3fa0 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56489d8aa750 con 0x56489d99ee30 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f619384c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56489d8fdec0 con 0x56489d99e440 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f61837fe640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 0x7f61780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35426/0 (socket says 172.21.15.107:35426) 2024-04-24T23:42:58.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 -- 172.21.15.107:0/2924204447 learned_addr learned my addr 172.21.15.107:0/2924204447 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f618ad9d640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 0x7f61780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d99e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f618a59c640 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f61837fe640 1 -- 172.21.15.107:0/2405555209 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f617c0efc30 con 0x7f618c101920 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 -- 172.21.15.107:0/2924204447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 msgr2=0x56489d99e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d99e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 -- 172.21.15.107:0/2924204447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 msgr2=0x56489d9a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.035+0000 7f6189d9b640 1 -- 172.21.15.107:0/2924204447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489d6162d0 con 0x56489d9a3fa0 2024-04-24T23:42:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f6189d9b640 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6174009ef0 tx=0x7f6174009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27935369d470983 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f618ad9d640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 0x7f61780a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f617c0bc1e0 tx=0x7f617c0872c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f6180ff9640 1 -- 172.21.15.107:0/2924204447 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174017470 con 0x56489d9a3fa0 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f6180ff9640 1 -- 172.21.15.107:0/2924204447 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617401c070 con 0x56489d9a3fa0 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f6180ff9640 1 -- 172.21.15.107:0/2924204447 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61740159a0 con 0x56489d9a3fa0 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 -- 172.21.15.107:0/2924204447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 msgr2=0x56489d9a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6174009ef0 tx=0x7f6174009580 comp rx=0 tx=0).stop 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 -- 172.21.15.107:0/2924204447 shutdown_connections 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d9a3fa0 0x56489d9a6430 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 --2- 172.21.15.107:0/2924204447 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d99e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.036+0000 7f619384c740 1 -- 172.21.15.107:0/2924204447 >> 172.21.15.107:0/2924204447 conn(0x56489d999ab0 msgr2=0x56489d999eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- 172.21.15.107:0/2924204447 shutdown_connections 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- 172.21.15.107:0/2924204447 wait complete. 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- start start 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 Processor -- start 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d9a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d9a3fa0 0x56489d9afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56489d9a6f50 con 0x56489d99ee30 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56489d8bdcf0 con 0x56489d9a3fa0 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56489d9a70d0 con 0x56489d99e440 2024-04-24T23:42:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f619384c740 10 client.0 _ll_register_callbacks cb 0x56489d8bd6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f618a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f618a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35428/0 (socket says 172.21.15.107:35428) 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f618a59c640 1 -- 172.21.15.107:0/608133041 learned_addr learned my addr 172.21.15.107:0/608133041 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f6189d9b640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d9a3fa0 0x56489d9afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f6189d9b640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d9a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f618a59c640 1 -- 172.21.15.107:0/608133041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 msgr2=0x56489d9a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.037+0000 7f618a59c640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d9a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f618a59c640 1 -- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d9a3fa0 msgr2=0x56489d9afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f618a59c640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d9a3fa0 0x56489d9afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f618a59c640 1 -- 172.21.15.107:0/608133041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489d6e06b0 con 0x56489d99ee30 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6189d9b640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d9a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f618a59c640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6184010700 tx=0x7f6184010bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dae67c81387629f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184003b70 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6184003d10 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f617401c070 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f618400e870 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 10 client.14772 fetch_fsmap learned FSMap version 17 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56489d9b2ef0 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617401c070 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f618400f040 con 0x56489d99ee30 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 10 client.14772 fetch_fsmap finished waiting for FSMap version 17 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 20 client.14772 populate_metadata read hostname 'smithi107' 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489d9b5640 0x56489d9b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 4 client.14772 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:42:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.038+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56489d90f440 con 0x56489d9b5640 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.039+0000 7f6189d9b640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489d9b5640 0x56489d9b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.039+0000 7f6171ffb640 10 client.14772 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.039+0000 7f6189d9b640 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489d9b5640 0x56489d9b6ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe20fbae568b331 server_cookie=6684679fe34e4401 in_seq=0 out_seq=0 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f6171ffb640 1 -- 172.21.15.107:0/608133041 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56489d90f440 con 0x56489d9b5640 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f6171ffb640 10 client.14772 handle_command_reply: tid=0 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 client.14772 shutdown 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 msgr2=0x56489d9a8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6184010700 tx=0x7f6184010bc0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 shutdown_connections 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489d9b5640 0x56489d9b6ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d9a3fa0 0x56489d9afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9a8db0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f6171ffb640 0 client.14772 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:42:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 --2- 172.21.15.107:0/608133041 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d99e440 0x56489d9a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 >> 172.21.15.107:0/608133041 conn(0x56489d999ab0 msgr2=0x56489d998ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 shutdown_connections 2024-04-24T23:42:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 1 -- 172.21.15.107:0/608133041 wait complete. 2024-04-24T23:42:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.040+0000 7f619384c740 20 client.14772 trim_cache size 0 max 16384 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 Processor -- start 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 -- start start 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489d9a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b4df0 con 0x56489d99e440 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b54c0 con 0x56489d99ee30 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f619384c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b5b90 con 0x56489d9a3fa0 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35434/0 (socket says 172.21.15.107:35434) 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 -- 172.21.15.107:0/3873579132 learned_addr learned my addr 172.21.15.107:0/3873579132 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f618a59c640 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 -- 172.21.15.107:0/3873579132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 msgr2=0x56489d9a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489d9a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 -- 172.21.15.107:0/3873579132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 msgr2=0x56489d9afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 -- 172.21.15.107:0/3873579132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489d99f210 con 0x56489d99e440 2024-04-24T23:42:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f618a59c640 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.058+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f61740156f0 tx=0x7f6174015440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398b3eb463573c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f6173fff640 1 -- 172.21.15.107:0/3873579132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617402f070 con 0x56489d99e440 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f6173fff640 1 -- 172.21.15.107:0/3873579132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617401c070 con 0x56489d99e440 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f6173fff640 1 -- 172.21.15.107:0/3873579132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174015cd0 con 0x56489d99e440 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- 172.21.15.107:0/3873579132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 msgr2=0x56489d9a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f61740156f0 tx=0x7f6174015440 comp rx=0 tx=0).stop 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- 172.21.15.107:0/3873579132 shutdown_connections 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489d9a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99ee30 0x56489d9afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 --2- 172.21.15.107:0/3873579132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99e440 0x56489d9a6390 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- 172.21.15.107:0/3873579132 >> 172.21.15.107:0/3873579132 conn(0x56489d999ab0 msgr2=0x56489d994690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- 172.21.15.107:0/3873579132 shutdown_connections 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- 172.21.15.107:0/3873579132 wait complete. 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.059+0000 7f619384c740 1 -- start start 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 Processor -- start 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 0x56489d9aad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489da195e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b6930 con 0x56489d99ee30 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b6260 con 0x56489d99e440 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56489d9b7000 con 0x56489d9a3fa0 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 10 client.0 _ll_register_callbacks cb 0x7f6184009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35436/0 (socket says 172.21.15.107:35436) 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 -- 172.21.15.107:0/3803534134 learned_addr learned my addr 172.21.15.107:0/3803534134 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 0x56489d9aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 -- 172.21.15.107:0/3803534134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 msgr2=0x56489da195e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489da195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 -- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 msgr2=0x56489d9aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 0x56489d9aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 -- 172.21.15.107:0/3803534134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489da19b50 con 0x56489d99ee30 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 0x56489d9aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f618a59c640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6184006010 tx=0x7f618400e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ead649638a4546 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618400ec10 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f618400edb0 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f617401c070 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f61840129e0 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.060+0000 7f619384c740 10 client.14778 fetch_fsmap learned FSMap version 17 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56489da1adf0 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617401c070 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f618402b420 con 0x56489d99ee30 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 10 client.14778 fetch_fsmap finished waiting for FSMap version 17 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 10 client.14778 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 20 client.14778 populate_metadata read hostname 'smithi107' 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489da1fb00 0x56489da1e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 4 client.14778 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:42:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "status"}) -- 0x56489d909790 con 0x56489da1fb00 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.061+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489da1fb00 0x56489da1e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f6171ffb640 10 client.14778 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f6189d9b640 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489da1fb00 0x56489da1e810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24639ba8fd73227a server_cookie=7e2119c465c2c9ec in_seq=0 out_seq=0 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f6171ffb640 1 -- 172.21.15.107:0/3803534134 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56489d909790 con 0x56489da1fb00 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f6171ffb640 10 client.14778 handle_command_reply: tid=0 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 client.14778 shutdown 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 msgr2=0x56489d9ab240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6184006010 tx=0x7f618400e800 comp rx=0 tx=0).stop 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 shutdown_connections 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56489da1fb00 0x56489da1e810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56489d9a3fa0 0x56489da195e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f6171ffb640 0 client.14778 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:42:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56489d99ee30 0x56489d9ab240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 --2- 172.21.15.107:0/3803534134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56489d99e440 0x56489d9aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 >> 172.21.15.107:0/3803534134 conn(0x56489d999ab0 msgr2=0x56489d9a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 shutdown_connections 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 1 -- 172.21.15.107:0/3803534134 wait complete. 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.062+0000 7f619384c740 20 client.14778 trim_cache size 0 max 16384 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-04-24T23:42:58.071 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24506, 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.7798240500000002, 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 89, 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 13.654342245 2024-04-24T23:42:58.072 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 msgr2=0x7f61780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 0x7f61780a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f617c0bc1e0 tx=0x7f617c0872c0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 msgr2=0x7f618c19ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f617c0314e0 tx=0x7f617c004360 comp rx=0 tx=0).stop 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 shutdown_connections 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f61780a2b90 0x7f61780a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f618c101920 0x7f618c19ee40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f618c100f60 0x7f618c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 --2- 172.21.15.107:0/2405555209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f618c0ffe10 0x7f618c1097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 >> 172.21.15.107:0/2405555209 conn(0x7f618c0fb380 msgr2=0x7f618c104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 shutdown_connections 2024-04-24T23:42:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:58.063+0000 7f6190b48640 1 -- 172.21.15.107:0/2405555209 wait complete. 2024-04-24T23:42:58.075 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:42:58.075 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:42:58.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-24T23:42:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:42:58.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:58 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2632224803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:58 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/456302734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:58 smithi170 ceph-mon[32714]: pgmap v294: 129 pgs: 1 creating+peering, 20 unknown, 108 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-24T23:42:58.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:58 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2632224803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:58 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/456302734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:58 smithi107 ceph-mon[29118]: pgmap v294: 129 pgs: 1 creating+peering, 20 unknown, 108 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-24T23:42:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:58 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2632224803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:58 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/456302734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:42:58.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:58 smithi195 ceph-mon[33019]: pgmap v294: 129 pgs: 1 creating+peering, 20 unknown, 108 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-24T23:42:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:59 smithi170 ceph-mon[32714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:42:59 smithi170 ceph-mon[32714]: Cluster is now healthy 2024-04-24T23:42:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:59 smithi107 ceph-mon[29118]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:59.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:42:59 smithi107 ceph-mon[29118]: Cluster is now healthy 2024-04-24T23:42:59.713 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:59 smithi195 ceph-mon[33019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T23:42:59.713 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:42:59 smithi195 ceph-mon[33019]: Cluster is now healthy 2024-04-24T23:42:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 -- 172.21.15.107:0/2250389772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc101ce0 msgr2=0x7f0adc102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/2250389772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc101ce0 0x7f0adc102160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80099c0 tx=0x7f0ad8009cd0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 -- 172.21.15.107:0/2250389772 shutdown_connections 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/2250389772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc1026a0 0x7f0adc10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/2250389772 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 0x7f0adc1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/2250389772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc101ce0 0x7f0adc102160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 -- 172.21.15.107:0/2250389772 >> 172.21.15.107:0/2250389772 conn(0x7f0adc076450 msgr2=0x7f0adc076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 -- 172.21.15.107:0/2250389772 shutdown_connections 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.982+0000 7f0ae3c85700 1 -- 172.21.15.107:0/2250389772 wait complete. 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 Processor -- start 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 -- start start 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 0x7f0adc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 0x7f0adc1a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc111660 con 0x7f0adc1026a0 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1114e0 con 0x7f0adc101ce0 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae3c85700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1117e0 con 0x7f0adc107400 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35452/0 (socket says 172.21.15.107:35452) 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 -- 172.21.15.107:0/435585779 learned_addr learned my addr 172.21.15.107:0/435585779 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae2222700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 0x7f0adc1a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1a21700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 0x7f0adc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 -- 172.21.15.107:0/435585779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 msgr2=0x7f0adc1a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 0x7f0adc1a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 -- 172.21.15.107:0/435585779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 msgr2=0x7f0adc1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 0x7f0adc1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 -- 172.21.15.107:0/435585779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc1a4fa0 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1a21700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 0x7f0adc1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.983+0000 7f0ae1220700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009990 tx=0x7f0ad8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.984+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8026070 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.984+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ad8003780 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.984+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800fc90 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.984+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0adc1a5170 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.984+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0adc1a5620 con 0x7f0adc1026a0 2024-04-24T23:42:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.985+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0adc068f20 con 0x7f0adc1026a0 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.986+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0ad8029070 con 0x7f0adc1026a0 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.986+0000 7f0ad2ffd700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 0x7f0ac80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.986+0000 7f0ae1a21700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 0x7f0ac80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.987+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0ad80cffb0 con 0x7f0adc1026a0 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.987+0000 7f0ae1a21700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 0x7f0ac80a5160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005fd0 tx=0x7f0acc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:42:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:42:59.990+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ad8030020 con 0x7f0adc1026a0 2024-04-24T23:43:00.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.230+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0adc02db30 con 0x7f0adc1026a0 2024-04-24T23:43:00.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.231+0000 7f0ad2ffd700 1 -- 172.21.15.107:0/435585779 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f0ad8013070 con 0x7f0adc1026a0 2024-04-24T23:43:00.231 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:43:00.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 msgr2=0x7f0ac80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:00.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 0x7f0ac80a5160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005fd0 tx=0x7f0acc005ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 msgr2=0x7f0adc199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009990 tx=0x7f0ad8009f80 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 shutdown_connections 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0ac80a2ca0 0x7f0ac80a5160 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0adc107400 0x7f0adc1a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0adc101ce0 0x7f0adc1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 --2- 172.21.15.107:0/435585779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0adc1026a0 0x7f0adc199f20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 >> 172.21.15.107:0/435585779 conn(0x7f0adc076450 msgr2=0x7f0adc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 shutdown_connections 2024-04-24T23:43:00.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00.233+0000 7f0ae3c85700 1 -- 172.21.15.107:0/435585779 wait complete. 2024-04-24T23:43:00.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:00 smithi107 ceph-mon[29118]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-24T23:43:00.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:00 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/435585779' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:43:00.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:00 smithi170 ceph-mon[32714]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-24T23:43:00.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:00 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/435585779' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:43:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:00 smithi195 ceph-mon[33019]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-24T23:43:00.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:00 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/435585779' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:43:00.884 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:43:00.885 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:43:00.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:00.885 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-04-24T23:43:00.902 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:43:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:43:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:43:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-04-24T23:43:00.903 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-24T23:43:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42227sec preferred_lft 42227sec 2024-04-24T23:43:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::6f34:b7ae:a569:134c/64 scope link noprefixroute 2024-04-24T23:43:00.904 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:43:00.904 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:43:00.905 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:43:00.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:43:01.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:01.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:01.098 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:43:01.122 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-04-24T23:43:01.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:01.124 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-04-24T23:43:01.197 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-04-24T23:43:01.197 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:43:01.197 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T23:43:01.197 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T23:43:01.197 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:43:01.198 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:43:01.198 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:43:01.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T23:43:01.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T23:43:01.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T23:43:01.199 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:43:01.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:43:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:01.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:01.505 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-04-24T23:43:01.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:01.520 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-04-24T23:43:01.574 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:43:01.574 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:43:01.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:43:01.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-24T23:43:01.575 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:43:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:43:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:01.690 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-24T23:43:01.690 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:43:01.690 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:43:01.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-24T23:43:01.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:43:01.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-24T23:43:01.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-24T23:43:01.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-24T23:43:01.691 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:43:01.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:43:01.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:01.827 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:43:01.827 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:43:01.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-04-24T23:43:01.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:43:01.827 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:43:01.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:43:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:01.901 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-24T23:43:01.901 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:43:01.901 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:01.916 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:43:01.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:43:01.916 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:01.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-04-24T23:43:02.036 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-04-24T23:43:02.093 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:43:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/562d47ca1115c247165bf632b4bb6bdfedca8e388039afb63a953a9908d0f8ad/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/e4cc50a62c2949988e79b1cdbaad62a9badd664c12027e05f56c682023167cc4/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/0377c886ca60f1d6bb708d8abf7ea58b3a684351f45d29728c3117cb6b1fbb72/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bb18a5156cd6ca15c0d9d7aea3919fc20f988b6ffd310d24d837e23eec0229e3/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/fdd6abc7ee48c761300bfc516c2dac31518b0ffe00e1fb510928c9b5dd3433ab/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/762b5c129e896c5c5a3718ba906996649e61da58da0322526d3abff5f0f52e02/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/878fff6685be4b00b3a35c4bd09c1f3e2bdf00a26da7454317d979c39871f4f0/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/56183285ebb5e992bf0421d6447f2ff99c671f2e0ae0f33d66641ed5b26a9ebd/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3c58ba5a3cf44d3add68075ee81bffac5abdd4d72388e085b98c8d7c3eb310df/merged 2024-04-24T23:43:02.095 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b59c97041e0a21d138b655b253fbfe43ff90f1087c2c5afba553df848c3eeeca/merged 2024-04-24T23:43:02.096 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-04-24T23:43:02.096 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:43:02.096 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:43:02.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:02.096 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-04-24T23:43:02.149 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:43:02.150 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-24T23:43:02.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-04-24T23:43:02.215 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-04-24T23:43:02.270 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-04-24T23:43:02.271 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-04-24T23:43:02.272 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/562d47ca1115c247165bf632b4bb6bdfedca8e388039afb63a953a9908d0f8ad/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/e4cc50a62c2949988e79b1cdbaad62a9badd664c12027e05f56c682023167cc4/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/0377c886ca60f1d6bb708d8abf7ea58b3a684351f45d29728c3117cb6b1fbb72/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bb18a5156cd6ca15c0d9d7aea3919fc20f988b6ffd310d24d837e23eec0229e3/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/fdd6abc7ee48c761300bfc516c2dac31518b0ffe00e1fb510928c9b5dd3433ab/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/762b5c129e896c5c5a3718ba906996649e61da58da0322526d3abff5f0f52e02/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/878fff6685be4b00b3a35c4bd09c1f3e2bdf00a26da7454317d979c39871f4f0/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/56183285ebb5e992bf0421d6447f2ff99c671f2e0ae0f33d66641ed5b26a9ebd/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3c58ba5a3cf44d3add68075ee81bffac5abdd4d72388e085b98c8d7c3eb310df/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b59c97041e0a21d138b655b253fbfe43ff90f1087c2c5afba553df848c3eeeca/merged 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:43:02.273 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:43:02.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:02.274 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-04-24T23:43:02.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:2024-04-24T23:43:02.292+0000 7f22c4635480 -1 init, newargv = 0x55f086791080 newargc=15 2024-04-24T23:43:02.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[70912]: starting ceph client 2024-04-24T23:43:02.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[70912]: starting fuse 2024-04-24T23:43:02.306 INFO:teuthology.orchestra.run.smithi107.stdout:115 2024-04-24T23:43:02.306 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> import re 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> import os 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2024-04-24T23:43:02.307 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> return f 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.admin")) 2024-04-24T23:43:02.308 DEBUG:teuthology.orchestra.run.smithi107:> ' 2024-04-24T23:43:02.423 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.admin.70912.asok 2024-04-24T23:43:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:02.433 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.70912.asok 2024-04-24T23:43:02.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:02.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.70912.asok status 2024-04-24T23:43:02.566 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "admin", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "70912", 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/" 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 14790, 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2024-04-24T23:43:02.567 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 14790 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1190134575 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1190134575 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:43:02.568 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.14790 192.168.0.1:0/1190134575", 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/1190134575", 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 1, 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 17, 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 89, 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs" 2024-04-24T23:43:02.569 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:43:02.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:02.575 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2024-04-24T23:43:02.575 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:02.590 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2024-04-24T23:43:02.590 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:02.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:02.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:02.655 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-24T23:43:02.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:43:02.770 INFO:teuthology.orchestra.run.smithi107.stdout:# file: . 2024-04-24T23:43:02.770 INFO:teuthology.orchestra.run.smithi107.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-24T23:43:02.770 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:02.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:43:02.775 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:43:02.776 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:43:02.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:43:02.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- 172.21.15.107:0/2319727467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a4102ac0 msgr2=0x7fa9a4102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 --2- 172.21.15.107:0/2319727467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4102f40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009930 tx=0x7fa98c02a420 comp rx=0 tx=0).stop 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- 172.21.15.107:0/2319727467 shutdown_connections 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 --2- 172.21.15.107:0/2319727467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4103480 0x7fa9a4109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 --2- 172.21.15.107:0/2319727467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4102f40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 --2- 172.21.15.107:0/2319727467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a40fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- 172.21.15.107:0/2319727467 >> 172.21.15.107:0/2319727467 conn(0x7fa9a40fb420 msgr2=0x7fa9a40fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- 172.21.15.107:0/2319727467 shutdown_connections 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- 172.21.15.107:0/2319727467 wait complete. 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 Processor -- start 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.930+0000 7fa9a897e640 1 -- start start 2024-04-24T23:43:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a4103480 0x7fa9a419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d5c0 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d440 con 0x7fa9a4103480 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d740 con 0x7fa9a4102ac0 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35456/0 (socket says 172.21.15.107:35456) 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 -- 172.21.15.107:0/3807753682 learned_addr learned my addr 172.21.15.107:0/3807753682 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a1d9b640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a2d9d640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a4103480 0x7fa9a419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 -- 172.21.15.107:0/3807753682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4102ac0 msgr2=0x7fa9a4198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 -- 172.21.15.107:0/3807753682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a4103480 msgr2=0x7fa9a419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a4103480 0x7fa9a419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 -- 172.21.15.107:0/3807753682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a419f860 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a259c640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00b450 tx=0x7fa99c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c015020 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c011d70 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a419fb50 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00f340 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.931+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a41a00a0 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.932+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a4108bf0 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.933+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa99c018020 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.934+0000 7fa99b7fe640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 0x7fa9900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.934+0000 7fa9a1d9b640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 0x7fa9900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.934+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa99c0cee20 con 0x7fa9a40ffaf0 2024-04-24T23:43:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.934+0000 7fa9a1d9b640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 0x7fa9900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa98c006010 tx=0x7fa98c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:02.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:02.939+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa99c01f020 con 0x7fa9a40ffaf0 2024-04-24T23:43:03.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.118+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9a4062340 con 0x7fa9a40ffaf0 2024-04-24T23:43:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.119+0000 7fa99b7fe640 1 -- 172.21.15.107:0/3807753682 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7fa99c00d040 con 0x7fa9a40ffaf0 2024-04-24T23:43:03.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:03.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 msgr2=0x7fa9900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 0x7fa9900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa98c006010 tx=0x7fa98c000bc0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 msgr2=0x7fa9a4197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00b450 tx=0x7fa99c00b910 comp rx=0 tx=0).stop 2024-04-24T23:43:03.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 shutdown_connections 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9900a2b90 0x7fa9900a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a4103480 0x7fa9a419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa9a4102ac0 0x7fa9a4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 --2- 172.21.15.107:0/3807753682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a40ffaf0 0x7fa9a4197c50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 >> 172.21.15.107:0/3807753682 conn(0x7fa9a40fb420 msgr2=0x7fa9a4105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 shutdown_connections 2024-04-24T23:43:03.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.121+0000 7fa9a897e640 1 -- 172.21.15.107:0/3807753682 wait complete. 2024-04-24T23:43:03.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:43:03.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:43:03.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 -- 172.21.15.107:0/624441588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b840ffa50 msgr2=0x7f5b840ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/624441588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b840ffa50 0x7f5b840ffe70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009930 tx=0x7f5b6c02a420 comp rx=0 tx=0).stop 2024-04-24T23:43:03.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 -- 172.21.15.107:0/624441588 shutdown_connections 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/624441588 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 0x7f5b84109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/624441588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b84102a20 0x7f5b84102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/624441588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b840ffa50 0x7f5b840ffe70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 -- 172.21.15.107:0/624441588 >> 172.21.15.107:0/624441588 conn(0x7f5b840fb380 msgr2=0x7f5b840fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 -- 172.21.15.107:0/624441588 shutdown_connections 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.235+0000 7f5b88ab2640 1 -- 172.21.15.107:0/624441588 wait complete. 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 Processor -- start 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 -- start start 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 0x7f5b84197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 0x7f5b8419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8410d4d0 con 0x7f5b84102a20 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8410d350 con 0x7f5b840ffa50 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b88ab2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8410d650 con 0x7f5b841033e0 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35480/0 (socket says 172.21.15.107:35480) 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 -- 172.21.15.107:0/1533826633 learned_addr learned my addr 172.21.15.107:0/1533826633 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b8259c640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 0x7f5b84197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b82d9d640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 0x7f5b8419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 -- 172.21.15.107:0/1533826633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 msgr2=0x7f5b8419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 0x7f5b8419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 -- 172.21.15.107:0/1533826633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 msgr2=0x7f5b84197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 0x7f5b84197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 -- 172.21.15.107:0/1533826633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8419f710 con 0x7f5b84102a20 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b8259c640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 0x7f5b84197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:03.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b81d9b640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400c920 tx=0x7f5b7400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.236+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74007c70 con 0x7f5b84102a20 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.237+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b74007e10 con 0x7f5b84102a20 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.237+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8419fa00 con 0x7f5b84102a20 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.237+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74018890 con 0x7f5b84102a20 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.237+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8419ffd0 con 0x7f5b84102a20 2024-04-24T23:43:03.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.238+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b84108b50 con 0x7f5b84102a20 2024-04-24T23:43:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.239+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5b74018a70 con 0x7f5b84102a20 2024-04-24T23:43:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.239+0000 7f5b7b7fe640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 0x7f5b700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.240+0000 7f5b8259c640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 0x7f5b700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.240+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5b740cf480 con 0x7f5b84102a20 2024-04-24T23:43:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.240+0000 7f5b8259c640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 0x7f5b700a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009230 tx=0x7f5b6c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.244+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b74014030 con 0x7f5b84102a20 2024-04-24T23:43:03.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:03 smithi195 ceph-mon[33019]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 533 B/s wr, 1 op/s 2024-04-24T23:43:03.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:03 smithi170 ceph-mon[32714]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 533 B/s wr, 1 op/s 2024-04-24T23:43:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:03 smithi107 ceph-mon[29118]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 533 B/s wr, 1 op/s 2024-04-24T23:43:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.423+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b84062340 con 0x7f5b84102a20 2024-04-24T23:43:03.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.423+0000 7f5b7b7fe640 1 -- 172.21.15.107:0/1533826633 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f5b7400de20 con 0x7f5b84102a20 2024-04-24T23:43:03.423 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:43:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 msgr2=0x7f5b700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 0x7f5b700a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009230 tx=0x7f5b6c000bc0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 msgr2=0x7f5b84198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400c920 tx=0x7f5b7400cde0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 shutdown_connections 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5b700a2b90 0x7f5b700a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5b841033e0 0x7f5b8419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84102a20 0x7f5b84198030 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 --2- 172.21.15.107:0/1533826633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b840ffa50 0x7f5b84197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 >> 172.21.15.107:0/1533826633 conn(0x7f5b840fb380 msgr2=0x7f5b84105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 shutdown_connections 2024-04-24T23:43:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.425+0000 7f5b88ab2640 1 -- 172.21.15.107:0/1533826633 wait complete. 2024-04-24T23:43:03.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:43:03.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 -- 172.21.15.107:0/672153437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 msgr2=0x7f93800ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 --2- 172.21.15.107:0/672153437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f93800ffe70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9370009930 tx=0x7f937002a420 comp rx=0 tx=0).stop 2024-04-24T23:43:03.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 -- 172.21.15.107:0/672153437 shutdown_connections 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 --2- 172.21.15.107:0/672153437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 0x7f9380109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 --2- 172.21.15.107:0/672153437 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 --2- 172.21.15.107:0/672153437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f93800ffe70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 -- 172.21.15.107:0/672153437 >> 172.21.15.107:0/672153437 conn(0x7f93800fb380 msgr2=0x7f93800fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 -- 172.21.15.107:0/672153437 shutdown_connections 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.537+0000 7f9384929640 1 -- 172.21.15.107:0/672153437 wait complete. 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 Processor -- start 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 -- start start 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 0x7f938019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f938010d480 con 0x7f93800ffa50 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f938010d300 con 0x7f93801033e0 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f9384929640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f938010d600 con 0x7f9380102a20 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35494/0 (socket says 172.21.15.107:35494) 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 -- 172.21.15.107:0/3378590172 learned_addr learned my addr 172.21.15.107:0/3378590172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937ed9d640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 0x7f938019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937dd9b640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 -- 172.21.15.107:0/3378590172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 msgr2=0x7f9380198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 -- 172.21.15.107:0/3378590172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 msgr2=0x7f938019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 0x7f938019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937e59c640 1 -- 172.21.15.107:0/3378590172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938019f710 con 0x7f93800ffa50 2024-04-24T23:43:03.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.538+0000 7f937dd9b640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f937e59c640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9370031440 tx=0x7f93700045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937003f030 con 0x7f93800ffa50 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938019f9a0 con 0x7f93800ffa50 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9370003ce0 con 0x7f93800ffa50 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9370035070 con 0x7f93800ffa50 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.539+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938019ff60 con 0x7f93800ffa50 2024-04-24T23:43:03.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.540+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9380108bd0 con 0x7f93800ffa50 2024-04-24T23:43:03.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.541+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9370042030 con 0x7f93800ffa50 2024-04-24T23:43:03.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.541+0000 7f936f7fe640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 0x7f93680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.541+0000 7f937dd9b640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 0x7f93680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.541+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93700ef960 con 0x7f93800ffa50 2024-04-24T23:43:03.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.542+0000 7f937dd9b640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 0x7f93680a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9380199000 tx=0x7f9374005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.546+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f93700bc680 con 0x7f93800ffa50 2024-04-24T23:43:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.733+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9380062340 con 0x7f93800ffa50 2024-04-24T23:43:03.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.734+0000 7f936f7fe640 1 -- 172.21.15.107:0/3378590172 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f937003b460 con 0x7f93800ffa50 2024-04-24T23:43:03.734 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 msgr2=0x7f93680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 0x7f93680a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9380199000 tx=0x7f9374005d90 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 msgr2=0x7f9380197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9370031440 tx=0x7f93700045e0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 shutdown_connections 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93680a2b90 0x7f93680a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93801033e0 0x7f938019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9380102a20 0x7f9380198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 --2- 172.21.15.107:0/3378590172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93800ffa50 0x7f9380197af0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 >> 172.21.15.107:0/3378590172 conn(0x7f93800fb380 msgr2=0x7f9380105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 shutdown_connections 2024-04-24T23:43:03.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.736+0000 7f9384929640 1 -- 172.21.15.107:0/3378590172 wait complete. 2024-04-24T23:43:03.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:43:03.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 -- 172.21.15.107:0/1405443332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4102200 msgr2=0x7fc1b41045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/1405443332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4102200 0x7fc1b41045f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009900 tx=0x7fc1a402a410 comp rx=0 tx=0).stop 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 -- 172.21.15.107:0/1405443332 shutdown_connections 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/1405443332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 0x7fc1b41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/1405443332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4104b30 0x7fc1b4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/1405443332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4102200 0x7fc1b41045f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 -- 172.21.15.107:0/1405443332 >> 172.21.15.107:0/1405443332 conn(0x7fc1b40fbc30 msgr2=0x7fc1b40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 -- 172.21.15.107:0/1405443332 shutdown_connections 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.847+0000 7fc1b9d41640 1 -- 172.21.15.107:0/1405443332 wait complete. 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 Processor -- start 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 -- start start 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4102200 0x7fc1b41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 0x7fc1b419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b410d400 con 0x7fc1b4104b30 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b410d280 con 0x7fc1b4107460 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b9d41640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b410d580 con 0x7fc1b4102200 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35514/0 (socket says 172.21.15.107:35514) 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 -- 172.21.15.107:0/2713259038 learned_addr learned my addr 172.21.15.107:0/2713259038 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b3fff640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 0x7fc1b419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b37fe640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4102200 0x7fc1b41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 -- 172.21.15.107:0/2713259038 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4102200 msgr2=0x7fc1b41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4102200 0x7fc1b41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 -- 172.21.15.107:0/2713259038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 msgr2=0x7fc1b419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 0x7fc1b419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:03.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.848+0000 7fc1b2ffd640 1 -- 172.21.15.107:0/2713259038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b419f600 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b2ffd640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c970 tx=0x7fc1a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8014070 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b419f8f0 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a8004440 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800f450 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.849+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b419fe40 con 0x7fc1b4104b30 2024-04-24T23:43:03.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.850+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b40fd720 con 0x7fc1b4104b30 2024-04-24T23:43:03.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.851+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc1a8017070 con 0x7fc1b4104b30 2024-04-24T23:43:03.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.851+0000 7fc1b0ff9640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 0x7fc19c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:03.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.852+0000 7fc1b37fe640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 0x7fc19c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:03.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.852+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1a80cf0c0 con 0x7fc1b4104b30 2024-04-24T23:43:03.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.852+0000 7fc1b37fe640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 0x7fc19c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000c00 tx=0x7fc1a4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:03.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:03.856+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc1a801e020 con 0x7fc1b4104b30 2024-04-24T23:43:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.032+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1b4062340 con 0x7fc1b4104b30 2024-04-24T23:43:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.034+0000 7fc1b0ff9640 1 -- 172.21.15.107:0/2713259038 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fc1a809bde0 con 0x7fc1b4104b30 2024-04-24T23:43:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:43:04.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.036+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 msgr2=0x7fc19c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.036+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 0x7fc19c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000c00 tx=0x7fc1a4031400 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.036+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 msgr2=0x7fc1b4197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.036+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c970 tx=0x7fc1a800ce30 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 shutdown_connections 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc19c0a2b90 0x7fc19c0a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b4107460 0x7fc1b419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1b4104b30 0x7fc1b4197f20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 --2- 172.21.15.107:0/2713259038 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1b4102200 0x7fc1b41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 >> 172.21.15.107:0/2713259038 conn(0x7fc1b40fbc30 msgr2=0x7fc1b4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 shutdown_connections 2024-04-24T23:43:04.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.037+0000 7fc1b9d41640 1 -- 172.21.15.107:0/2713259038 wait complete. 2024-04-24T23:43:04.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:43:04.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 -- 172.21.15.107:0/766255338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd8103350 msgr2=0x7f4bd8109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/766255338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd8103350 0x7f4bd8109be0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009ca0 tx=0x7f4bd002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 -- 172.21.15.107:0/766255338 shutdown_connections 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/766255338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd8103350 0x7f4bd8109be0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/766255338 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 0x7f4bd8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/766255338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd81017a0 0x7f4bd8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 -- 172.21.15.107:0/766255338 >> 172.21.15.107:0/766255338 conn(0x7f4bd80fcd10 msgr2=0x7f4bd80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 -- 172.21.15.107:0/766255338 shutdown_connections 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 -- 172.21.15.107:0/766255338 wait complete. 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.149+0000 7f4bdeb16640 1 Processor -- start 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 -- start start 2024-04-24T23:43:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 0x7f4bd8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd8103350 0x7f4bd819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d400 con 0x7f4bd81017a0 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d280 con 0x7f4bd8103350 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d580 con 0x7f4bd8102990 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35534/0 (socket says 172.21.15.107:35534) 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 -- 172.21.15.107:0/3715758537 learned_addr learned my addr 172.21.15.107:0/3715758537 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bd7fff640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 0x7f4bd8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdd0b3640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd8103350 0x7f4bd819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 -- 172.21.15.107:0/3715758537 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 msgr2=0x7f4bd8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 0x7f4bd8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 -- 172.21.15.107:0/3715758537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd8103350 msgr2=0x7f4bd819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd8103350 0x7f4bd819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 -- 172.21.15.107:0/3715758537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd819f5e0 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdc8b2640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c970 tx=0x7f4bcc00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc004220 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bcc0043c0 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd819f8d0 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.150+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00f450 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.151+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd819fe70 con 0x7f4bd81017a0 2024-04-24T23:43:04.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.152+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd8108a60 con 0x7f4bd81017a0 2024-04-24T23:43:04.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.152+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4bcc0079b0 con 0x7f4bd81017a0 2024-04-24T23:43:04.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.153+0000 7f4bd5ffb640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 0x7f4bc80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.153+0000 7f4bd7fff640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 0x7f4bc80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.153+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4bcc0cef80 con 0x7f4bd81017a0 2024-04-24T23:43:04.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.153+0000 7f4bd7fff640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 0x7f4bc80a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8198ed0 tx=0x7f4bc400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.157+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4bcc014030 con 0x7f4bd81017a0 2024-04-24T23:43:04.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3807753682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:04.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1533826633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3378590172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.333+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4bd802d170 con 0x7f4bd81017a0 2024-04-24T23:43:04.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.333+0000 7f4bd5ffb640 1 -- 172.21.15.107:0/3715758537 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f4bcc09bca0 con 0x7f4bd81017a0 2024-04-24T23:43:04.333 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:43:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 msgr2=0x7f4bc80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 0x7f4bc80a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8198ed0 tx=0x7f4bc400a2d0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 msgr2=0x7f4bd81979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c970 tx=0x7f4bcc00cd70 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 shutdown_connections 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bc80a2c10 0x7f4bc80a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bd8103350 0x7f4bd819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bd8102990 0x7f4bd8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 --2- 172.21.15.107:0/3715758537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd81017a0 0x7f4bd81979c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 >> 172.21.15.107:0/3715758537 conn(0x7f4bd80fcd10 msgr2=0x7f4bd8105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 shutdown_connections 2024-04-24T23:43:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.335+0000 7f4bdeb16640 1 -- 172.21.15.107:0/3715758537 wait complete. 2024-04-24T23:43:04.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3807753682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:04.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1533826633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3378590172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3807753682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:04.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1533826633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3378590172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:04.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:43:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 -- 172.21.15.107:0/1334447842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 msgr2=0x7f1158108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 --2- 172.21.15.107:0/1334447842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f1158108ae0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1150009ca0 tx=0x7f115002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 -- 172.21.15.107:0/1334447842 shutdown_connections 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 --2- 172.21.15.107:0/1334447842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f1158108ae0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 --2- 172.21.15.107:0/1334447842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1158101890 0x7f1158101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 --2- 172.21.15.107:0/1334447842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f11581006a0 0x7f1158100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 -- 172.21.15.107:0/1334447842 >> 172.21.15.107:0/1334447842 conn(0x7f11580fbc10 msgr2=0x7f11580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 -- 172.21.15.107:0/1334447842 shutdown_connections 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.445+0000 7f115ff70640 1 -- 172.21.15.107:0/1334447842 wait complete. 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 Processor -- start 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 -- start start 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11581006a0 0x7f115806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 0x7f115819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f115810c300 con 0x7f1158102250 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f115810c180 con 0x7f11581006a0 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115ff70640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f115810c480 con 0x7f1158101890 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35544/0 (socket says 172.21.15.107:35544) 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 -- 172.21.15.107:0/3975575221 learned_addr learned my addr 172.21.15.107:0/3975575221 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115dd0c640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11581006a0 0x7f115806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115d50b640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 0x7f115819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 -- 172.21.15.107:0/3975575221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 msgr2=0x7f115819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 0x7f115819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 -- 172.21.15.107:0/3975575221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11581006a0 msgr2=0x7f115806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11581006a0 0x7f115806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115e50d640 1 -- 172.21.15.107:0/3975575221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115819f690 con 0x7f1158102250 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.446+0000 7f115d50b640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 0x7f115819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f115e50d640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1150005e30 tx=0x7f1150031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150003c30 con 0x7f1158102250 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1150003dd0 con 0x7f1158102250 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115819f8c0 con 0x7f1158102250 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115002fde0 con 0x7f1158102250 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.447+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115819fe10 con 0x7f1158102250 2024-04-24T23:43:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.448+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11581079c0 con 0x7f1158102250 2024-04-24T23:43:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.449+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1150004050 con 0x7f1158102250 2024-04-24T23:43:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.449+0000 7f114effd640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 0x7f11440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.449+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11500f0700 con 0x7f1158102250 2024-04-24T23:43:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.449+0000 7f115dd0c640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 0x7f11440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.450+0000 7f115dd0c640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 0x7f11440a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11581016f0 tx=0x7f115400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.454+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f11500bd420 con 0x7f1158102250 2024-04-24T23:43:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.643+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1158062340 con 0x7f1158102250 2024-04-24T23:43:04.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.643+0000 7f114effd640 1 -- 172.21.15.107:0/3975575221 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+601 (secure 0 0 0) 0x7f11500bcd20 con 0x7f1158102250 2024-04-24T23:43:04.644 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:04.644 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:04.644 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4054350187 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4085406700 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/526392411 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1308114869 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2998673885 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:04.645 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2063300390 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:listed 10 entries 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 msgr2=0x7f11440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 0x7f11440a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11581016f0 tx=0x7f115400b390 comp rx=0 tx=0).stop 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 msgr2=0x7f115819ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1150005e30 tx=0x7f1150031ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 shutdown_connections 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11440a2b90 0x7f11440a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1158102250 0x7f115819ed80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1158101890 0x7f115819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 --2- 172.21.15.107:0/3975575221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11581006a0 0x7f115806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.645+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 >> 172.21.15.107:0/3975575221 conn(0x7f11580fbc10 msgr2=0x7f1158104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.646+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 shutdown_connections 2024-04-24T23:43:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.646+0000 7f115ff70640 1 -- 172.21.15.107:0/3975575221 wait complete. 2024-04-24T23:43:04.656 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:43:04.656 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:43:04.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:43:04.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.810+0000 7faa8866b640 1 -- 172.21.15.107:0/2574683732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 msgr2=0x7faa80100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.810+0000 7faa8866b640 1 --2- 172.21.15.107:0/2574683732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80100aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009900 tx=0x7faa7c02a410 comp rx=0 tx=0).stop 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- 172.21.15.107:0/2574683732 shutdown_connections 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- 172.21.15.107:0/2574683732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80102230 0x7faa80108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- 172.21.15.107:0/2574683732 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80101870 0x7faa80101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- 172.21.15.107:0/2574683732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80100aa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- 172.21.15.107:0/2574683732 >> 172.21.15.107:0/2574683732 conn(0x7faa800fbbf0 msgr2=0x7faa800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- 172.21.15.107:0/2574683732 shutdown_connections 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- 172.21.15.107:0/2574683732 wait complete. 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 Processor -- start 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- start start 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80101870 0x7faa80198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80102230 0x7faa8019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faa8010d420 con 0x7faa80100680 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faa8010d2a0 con 0x7faa80101870 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa8866b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7faa8010d5a0 con 0x7faa80102230 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa86407640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa86407640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35568/0 (socket says 172.21.15.107:35568) 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa86407640 1 -- 172.21.15.107:0/3342584150 learned_addr learned my addr 172.21.15.107:0/3342584150 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa86c08640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80102230 0x7faa8019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.811+0000 7faa85c06640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80101870 0x7faa80198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 -- 172.21.15.107:0/3342584150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80102230 msgr2=0x7faa8019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80102230 0x7faa8019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 -- 172.21.15.107:0/3342584150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80101870 msgr2=0x7faa80198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80101870 0x7faa80198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 -- 172.21.15.107:0/3342584150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8019f6f0 con 0x7faa80100680 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa86407640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faa7c031610 tx=0x7faa7c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c03f030 con 0x7faa80100680 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8019f980 con 0x7faa80100680 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7c003ad0 con 0x7faa80100680 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c035070 con 0x7faa80100680 2024-04-24T23:43:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.812+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8019fe70 con 0x7faa80100680 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.813+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa801079f0 con 0x7faa80100680 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.814+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7faa7c042030 con 0x7faa80100680 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.815+0000 7faa777fe640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 0x7faa6c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.815+0000 7faa85c06640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 0x7faa6c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.815+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faa7c0ef870 con 0x7faa80100680 2024-04-24T23:43:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.815+0000 7faa85c06640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 0x7faa6c0a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa80198fe0 tx=0x7faa70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.819+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa7c0bbee0 con 0x7faa80100680 2024-04-24T23:43:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.995+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa80062340 con 0x7faa80100680 2024-04-24T23:43:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.997+0000 7faa777fe640 1 -- 172.21.15.107:0/3342584150 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7faa7c0c01e0 con 0x7faa80100680 2024-04-24T23:43:04.997 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:04.997 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 msgr2=0x7faa6c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 0x7faa6c0a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa80198fe0 tx=0x7faa70005f90 comp rx=0 tx=0).stop 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 msgr2=0x7faa80197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faa7c031610 tx=0x7faa7c004270 comp rx=0 tx=0).stop 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 shutdown_connections 2024-04-24T23:43:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faa6c0a2c10 0x7faa6c0a50d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa80102230 0x7faa8019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa80101870 0x7faa80198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 --2- 172.21.15.107:0/3342584150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa80100680 0x7faa80197ad0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 >> 172.21.15.107:0/3342584150 conn(0x7faa800fbbf0 msgr2=0x7faa80104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 shutdown_connections 2024-04-24T23:43:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:04.999+0000 7faa8866b640 1 -- 172.21.15.107:0/3342584150 wait complete. 2024-04-24T23:43:05.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:43:05.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:43:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 -- 172.21.15.107:0/3308252036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac1034a0 msgr2=0x7fb6ac109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 --2- 172.21.15.107:0/3308252036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac109d30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009cd0 tx=0x7fb6a402a8e0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 -- 172.21.15.107:0/3308252036 shutdown_connections 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 --2- 172.21.15.107:0/3308252036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac109d30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 --2- 172.21.15.107:0/3308252036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 --2- 172.21.15.107:0/3308252036 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 0x7fb6ac0ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 -- 172.21.15.107:0/3308252036 >> 172.21.15.107:0/3308252036 conn(0x7fb6ac0fb380 msgr2=0x7fb6ac0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 -- 172.21.15.107:0/3308252036 shutdown_connections 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.116+0000 7fb6b4015640 1 -- 172.21.15.107:0/3308252036 wait complete. 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 Processor -- start 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 -- start start 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 0x7fb6ac197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac10d490 con 0x7fb6ac102ae0 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac10d310 con 0x7fb6ac1034a0 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b4015640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac10d610 con 0x7fb6ac0ffa70 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35584/0 (socket says 172.21.15.107:35584) 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 -- 172.21.15.107:0/1755288110 learned_addr learned my addr 172.21.15.107:0/1755288110 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b1db1640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 0x7fb6ac197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b25b2640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 -- 172.21.15.107:0/1755288110 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 msgr2=0x7fb6ac197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 0x7fb6ac197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 -- 172.21.15.107:0/1755288110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac1034a0 msgr2=0x7fb6ac19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.117+0000 7fb6b15b0640 1 -- 172.21.15.107:0/1755288110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac19f6c0 con 0x7fb6ac102ae0 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6b15b0640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c920 tx=0x7fb69c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007bb0 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb69c007d50 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac19f9b0 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0187d0 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.118+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac19ffd0 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.119+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ac108c10 con 0x7fb6ac102ae0 2024-04-24T23:43:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.120+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb69c00f450 con 0x7fb6ac102ae0 2024-04-24T23:43:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.120+0000 7fb6a2ffd640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.120+0000 7fb6b1db1640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.120+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb69c0cf360 con 0x7fb6ac102ae0 2024-04-24T23:43:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.121+0000 7fb6b1db1640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 0x7fb6980a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac069ac0 tx=0x7fb6a8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.125+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb69c014030 con 0x7fb6ac102ae0 2024-04-24T23:43:05.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.301+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6ac062340 con 0x7fb6ac102ae0 2024-04-24T23:43:05.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.302+0000 7fb6a2ffd640 1 -- 172.21.15.107:0/1755288110 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fb69c00de20 con 0x7fb6ac102ae0 2024-04-24T23:43:05.302 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:43:05.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.303+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 msgr2=0x7fb6980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.303+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 0x7fb6980a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac069ac0 tx=0x7fb6a8009110 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.303+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 msgr2=0x7fb6ac197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.303+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c920 tx=0x7fb69c00cd20 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 shutdown_connections 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6ac1034a0 0x7fb6ac19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6ac102ae0 0x7fb6ac197fe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 --2- 172.21.15.107:0/1755288110 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6ac0ffa70 0x7fb6ac197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 >> 172.21.15.107:0/1755288110 conn(0x7fb6ac0fb380 msgr2=0x7fb6ac105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 shutdown_connections 2024-04-24T23:43:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.304+0000 7fb6b4015640 1 -- 172.21.15.107:0/1755288110 wait complete. 2024-04-24T23:43:05.311 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:05 smithi195 ceph-mon[33019]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 472 B/s wr, 1 op/s 2024-04-24T23:43:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2713259038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3715758537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3975575221' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:05.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3342584150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:05.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:05 smithi107 ceph-mon[29118]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 472 B/s wr, 1 op/s 2024-04-24T23:43:05.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2713259038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3715758537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3975575221' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:05.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3342584150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:05.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:43:05.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:05 smithi170 ceph-mon[32714]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 472 B/s wr, 1 op/s 2024-04-24T23:43:05.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2713259038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3715758537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:05.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3975575221' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:05.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3342584150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 -- 172.21.15.107:0/1968633417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 msgr2=0x7fde801045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 --2- 172.21.15.107:0/1968633417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde801045f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009900 tx=0x7fde7c02a410 comp rx=0 tx=0).stop 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 -- 172.21.15.107:0/1968633417 shutdown_connections 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 --2- 172.21.15.107:0/1968633417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80107460 0x7fde801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 --2- 172.21.15.107:0/1968633417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80104b30 0x7fde80106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 --2- 172.21.15.107:0/1968633417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde801045f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 -- 172.21.15.107:0/1968633417 >> 172.21.15.107:0/1968633417 conn(0x7fde800fbbf0 msgr2=0x7fde800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 -- 172.21.15.107:0/1968633417 shutdown_connections 2024-04-24T23:43:05.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.423+0000 7fde87df4640 1 -- 172.21.15.107:0/1968633417 wait complete. 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 Processor -- start 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 -- start start 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 0x7fde80198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80107460 0x7fde8019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde8010d470 con 0x7fde80102200 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fde8010d2f0 con 0x7fde80107460 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde87df4640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fde8010d5f0 con 0x7fde80104b30 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35602/0 (socket says 172.21.15.107:35602) 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 -- 172.21.15.107:0/52689934 learned_addr learned my addr 172.21.15.107:0/52689934 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:05.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde86391640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80107460 0x7fde8019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde8538f640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 0x7fde80198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 -- 172.21.15.107:0/52689934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 msgr2=0x7fde80198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 0x7fde80198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 -- 172.21.15.107:0/52689934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80107460 msgr2=0x7fde8019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80107460 0x7fde8019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde85b90640 1 -- 172.21.15.107:0/52689934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8019f730 con 0x7fde80102200 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.424+0000 7fde8538f640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 0x7fde80198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde85b90640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009230 tx=0x7fde7c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c03f030 con 0x7fde80102200 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c003ce0 con 0x7fde80102200 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c035070 con 0x7fde80102200 2024-04-24T23:43:05.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8019f9c0 con 0x7fde80102200 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.425+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8019ff30 con 0x7fde80102200 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.426+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde800fd6e0 con 0x7fde80102200 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.427+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fde7c042030 con 0x7fde80102200 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.427+0000 7fde76ffd640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 0x7fde6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.428+0000 7fde8538f640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 0x7fde6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.428+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde7c0efaa0 con 0x7fde80102200 2024-04-24T23:43:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.428+0000 7fde8538f640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 0x7fde6c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde80199020 tx=0x7fde70005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.432+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde7c0bc7c0 con 0x7fde80102200 2024-04-24T23:43:05.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.606+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fde80062340 con 0x7fde80102200 2024-04-24T23:43:05.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.606+0000 7fde76ffd640 1 -- 172.21.15.107:0/52689934 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fde7c0bc0c0 con 0x7fde80102200 2024-04-24T23:43:05.606 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 msgr2=0x7fde6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 0x7fde6c0a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde80199020 tx=0x7fde70005d30 comp rx=0 tx=0).stop 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 msgr2=0x7fde80197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009230 tx=0x7fde7c004430 comp rx=0 tx=0).stop 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 shutdown_connections 2024-04-24T23:43:05.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fde6c0a2b90 0x7fde6c0a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde80107460 0x7fde8019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde80104b30 0x7fde80198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 --2- 172.21.15.107:0/52689934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde80102200 0x7fde80197b10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 >> 172.21.15.107:0/52689934 conn(0x7fde800fbbf0 msgr2=0x7fde80102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 shutdown_connections 2024-04-24T23:43:05.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.608+0000 7fde87df4640 1 -- 172.21.15.107:0/52689934 wait complete. 2024-04-24T23:43:05.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 -- 172.21.15.107:0/4154580758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 msgr2=0x7efc68100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 --2- 172.21.15.107:0/4154580758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc68100aa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efc64009900 tx=0x7efc6402a410 comp rx=0 tx=0).stop 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 -- 172.21.15.107:0/4154580758 shutdown_connections 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 --2- 172.21.15.107:0/4154580758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 0x7efc68108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 --2- 172.21.15.107:0/4154580758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 0x7efc68101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 --2- 172.21.15.107:0/4154580758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc68100aa0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 -- 172.21.15.107:0/4154580758 >> 172.21.15.107:0/4154580758 conn(0x7efc680fbc30 msgr2=0x7efc680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 -- 172.21.15.107:0/4154580758 shutdown_connections 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.721+0000 7efc70314640 1 -- 172.21.15.107:0/4154580758 wait complete. 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 Processor -- start 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 -- start start 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 0x7efc6819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 0x7efc6819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efc6810c350 con 0x7efc68100680 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efc6810c1d0 con 0x7efc68101870 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc70314640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7efc6810c4d0 con 0x7efc68102230 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35622/0 (socket says 172.21.15.107:35622) 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 -- 172.21.15.107:0/2553428489 learned_addr learned my addr 172.21.15.107:0/2553428489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e8b1640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 0x7efc6819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 -- 172.21.15.107:0/2553428489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 msgr2=0x7efc6819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6d8af640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 0x7efc6819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 0x7efc6819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 -- 172.21.15.107:0/2553428489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 msgr2=0x7efc6819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 0x7efc6819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 -- 172.21.15.107:0/2553428489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6819f720 con 0x7efc68100680 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.722+0000 7efc6e0b0640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efc64031610 tx=0x7efc640043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.723+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6403f030 con 0x7efc68100680 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.723+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc64003ce0 con 0x7efc68100680 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.723+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6819f950 con 0x7efc68100680 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.723+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64035070 con 0x7efc68100680 2024-04-24T23:43:05.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.723+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6819fec0 con 0x7efc68100680 2024-04-24T23:43:05.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.724+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc681079a0 con 0x7efc68100680 2024-04-24T23:43:05.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.725+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7efc64042030 con 0x7efc68100680 2024-04-24T23:43:05.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.725+0000 7efc5f7fe640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 0x7efc540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:05.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.725+0000 7efc6d8af640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 0x7efc540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:05.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.725+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efc640ef9e0 con 0x7efc68100680 2024-04-24T23:43:05.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.726+0000 7efc6d8af640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 0x7efc540a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efc6806a660 tx=0x7efc58005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:05.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.729+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc640bc700 con 0x7efc68100680 2024-04-24T23:43:05.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.904+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc68062340 con 0x7efc68100680 2024-04-24T23:43:05.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.904+0000 7efc5f7fe640 1 -- 172.21.15.107:0/2553428489 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7efc640bc000 con 0x7efc68100680 2024-04-24T23:43:05.904 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:43:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 msgr2=0x7efc540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 0x7efc540a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efc6806a660 tx=0x7efc58005cc0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 msgr2=0x7efc6806a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efc64031610 tx=0x7efc640043b0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 shutdown_connections 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efc540a2b90 0x7efc540a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efc68102230 0x7efc6819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc68101870 0x7efc6819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 --2- 172.21.15.107:0/2553428489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc68100680 0x7efc6806a8f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 >> 172.21.15.107:0/2553428489 conn(0x7efc680fbc30 msgr2=0x7efc68104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 shutdown_connections 2024-04-24T23:43:05.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:05.906+0000 7efc70314640 1 -- 172.21.15.107:0/2553428489 wait complete. 2024-04-24T23:43:05.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.018+0000 7fb14fe5d640 1 -- 172.21.15.107:0/3038397535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 msgr2=0x7fb148102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.018+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/3038397535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148102e00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb138009900 tx=0x7fb13802a410 comp rx=0 tx=0).stop 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- 172.21.15.107:0/3038397535 shutdown_connections 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/3038397535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb148109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/3038397535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148102e00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/3038397535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 0x7fb148101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- 172.21.15.107:0/3038397535 >> 172.21.15.107:0/3038397535 conn(0x7fb1480fcd20 msgr2=0x7fb1480ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- 172.21.15.107:0/3038397535 shutdown_connections 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- 172.21.15.107:0/3038397535 wait complete. 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 Processor -- start 2024-04-24T23:43:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- start start 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 0x7fb148197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb14819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb14810d3f0 con 0x7fb148102980 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb14810d270 con 0x7fb148103340 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14fe5d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb14810d570 con 0x7fb148101790 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14d3f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14d3f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35634/0 (socket says 172.21.15.107:35634) 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.019+0000 7fb14d3f8640 1 -- 172.21.15.107:0/265114985 learned_addr learned my addr 172.21.15.107:0/265114985 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14dbf9640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 0x7fb148197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14e3fa640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb14819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 -- 172.21.15.107:0/265114985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 msgr2=0x7fb148197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 0x7fb148197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 -- 172.21.15.107:0/265114985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 msgr2=0x7fb14819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb14819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 -- 172.21.15.107:0/265114985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14819f6f0 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14e3fa640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb14819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14d3f8640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb138009230 tx=0x7fb138004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13803f030 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14819f980 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb138003ce0 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138035070 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.020+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb14819fec0 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.021+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb148108bd0 con 0x7fb148102980 2024-04-24T23:43:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.022+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb138042030 con 0x7fb148102980 2024-04-24T23:43:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.023+0000 7fb13effd640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 0x7fb1340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.023+0000 7fb14dbf9640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 0x7fb1340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.023+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1380efc70 con 0x7fb148102980 2024-04-24T23:43:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.023+0000 7fb14dbf9640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 0x7fb1340a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1481027e0 tx=0x7fb144009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.027+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb1380bc990 con 0x7fb148102980 2024-04-24T23:43:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.206+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb148062340 con 0x7fb148102980 2024-04-24T23:43:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.207+0000 7fb13effd640 1 -- 172.21.15.107:0/265114985 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fb1380bc290 con 0x7fb148102980 2024-04-24T23:43:06.208 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:43:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 msgr2=0x7fb1340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 0x7fb1340a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1481027e0 tx=0x7fb144009720 comp rx=0 tx=0).stop 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 msgr2=0x7fb148198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb138009230 tx=0x7fb138004360 comp rx=0 tx=0).stop 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 shutdown_connections 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb1340a2b90 0x7fb1340a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb148103340 0x7fb14819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb148102980 0x7fb148198010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 --2- 172.21.15.107:0/265114985 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb148101790 0x7fb148197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 >> 172.21.15.107:0/265114985 conn(0x7fb1480fcd20 msgr2=0x7fb148105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 shutdown_connections 2024-04-24T23:43:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.209+0000 7fb14fe5d640 1 -- 172.21.15.107:0/265114985 wait complete. 2024-04-24T23:43:06.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1755288110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/52689934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2553428489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:43:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1755288110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/52689934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2553428489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- 172.21.15.107:0/3901392717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 msgr2=0x7f0894109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 --2- 172.21.15.107:0/3901392717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f0894109bd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f088c009ca0 tx=0x7f088c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- 172.21.15.107:0/3901392717 shutdown_connections 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 --2- 172.21.15.107:0/3901392717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f0894109bd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 --2- 172.21.15.107:0/3901392717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 0x7f0894102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 --2- 172.21.15.107:0/3901392717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f0894101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- 172.21.15.107:0/3901392717 >> 172.21.15.107:0/3901392717 conn(0x7f08940fcd20 msgr2=0x7f08940ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- 172.21.15.107:0/3901392717 shutdown_connections 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- 172.21.15.107:0/3901392717 wait complete. 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 Processor -- start 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.321+0000 7f089a3d5640 1 -- start start 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f08941979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 0x7f0894197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f089410d3f0 con 0x7f0894103340 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f089410d270 con 0x7f0894102980 2024-04-24T23:43:06.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f089a3d5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f089410d570 con 0x7f0894101790 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35658/0 (socket says 172.21.15.107:35658) 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 -- 172.21.15.107:0/4017678243 learned_addr learned my addr 172.21.15.107:0/4017678243 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f08937fe640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 0x7f0894197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0893fff640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f08941979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 -- 172.21.15.107:0/4017678243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 msgr2=0x7f08941979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f08941979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 -- 172.21.15.107:0/4017678243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 msgr2=0x7f0894197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 0x7f0894197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 -- 172.21.15.107:0/4017678243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089419f5f0 con 0x7f0894103340 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0893fff640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f08941979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:06.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f0898972640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f088c005e30 tx=0x7f088c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.322+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c003c30 con 0x7f0894103340 2024-04-24T23:43:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.323+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089419f880 con 0x7f0894103340 2024-04-24T23:43:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.323+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f088c003dd0 con 0x7f0894103340 2024-04-24T23:43:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.323+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c02fde0 con 0x7f0894103340 2024-04-24T23:43:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.323+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089419fdd0 con 0x7f0894103340 2024-04-24T23:43:06.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.323+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0894108a50 con 0x7f0894103340 2024-04-24T23:43:06.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.324+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f088c004050 con 0x7f0894103340 2024-04-24T23:43:06.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.325+0000 7f08917fa640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 0x7f087c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:06.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.325+0000 7f0893fff640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 0x7f087c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:06.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.325+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f088c0f0840 con 0x7f0894103340 2024-04-24T23:43:06.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.325+0000 7f0893fff640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 0x7f087c0a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f08941027e0 tx=0x7f0884005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:06.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.329+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f088c0bd680 con 0x7f0894103340 2024-04-24T23:43:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1755288110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/52689934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2553428489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:43:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.517+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0894062340 con 0x7f0894103340 2024-04-24T23:43:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.518+0000 7f08917fa640 1 -- 172.21.15.107:0/4017678243 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+601 (secure 0 0 0) 0x7f088c0bcf80 con 0x7f0894103340 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4054350187 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4085406700 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/526392411 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1308114869 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2998673885 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:43:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2063300390 2024-04-25T23:35:47.891427+0000 2024-04-24T23:43:06.519 INFO:teuthology.orchestra.run.smithi107.stderr:listed 10 entries 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 msgr2=0x7f087c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 0x7f087c0a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f08941027e0 tx=0x7f0884005e70 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 msgr2=0x7f089419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f088c005e30 tx=0x7f088c031ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 shutdown_connections 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f087c0a2b90 0x7f087c0a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0894103340 0x7f089419ef80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0894102980 0x7f0894197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 --2- 172.21.15.107:0/4017678243 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0894101790 0x7f08941979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 >> 172.21.15.107:0/4017678243 conn(0x7f08940fcd20 msgr2=0x7f0894105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 shutdown_connections 2024-04-24T23:43:06.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:06.520+0000 7f089a3d5640 1 -- 172.21.15.107:0/4017678243 wait complete. 2024-04-24T23:43:06.531 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2024-04-24T23:43:06.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:06.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:43:06.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[70912]: fuse finished with error 0 and tester_r 0 2024-04-24T23:43:06.555 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:43:07.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:07 smithi195 ceph-mon[33019]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 0 op/s 2024-04-24T23:43:07.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/265114985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:07.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4017678243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:07 smithi170 ceph-mon[32714]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 0 op/s 2024-04-24T23:43:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/265114985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:07.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4017678243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:07.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:07 smithi107 ceph-mon[29118]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 0 op/s 2024-04-24T23:43:07.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/265114985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:43:07.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4017678243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:43:09.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:09 smithi195 ceph-mon[33019]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 263 B/s wr, 0 op/s 2024-04-24T23:43:09.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:09 smithi170 ceph-mon[32714]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 263 B/s wr, 0 op/s 2024-04-24T23:43:09.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:09 smithi107 ceph-mon[29118]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 263 B/s wr, 0 op/s 2024-04-24T23:43:11.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:11 smithi195 ceph-mon[33019]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-24T23:43:11.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:11 smithi170 ceph-mon[32714]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-24T23:43:11.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:11 smithi107 ceph-mon[29118]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-24T23:43:12.658 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:43:12.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:43:12.658 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-24T23:43:12.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:43:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 -- 172.21.15.107:0/1719124183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 msgr2=0x7f4d741017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1719124183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741017e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009ca0 tx=0x7f4d6c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 -- 172.21.15.107:0/1719124183 shutdown_connections 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1719124183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741017e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1719124183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d741009c0 0x7f4d74100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1719124183 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d74100060 0x7f4d74100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 -- 172.21.15.107:0/1719124183 >> 172.21.15.107:0/1719124183 conn(0x7f4d740fbc20 msgr2=0x7f4d740fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 -- 172.21.15.107:0/1719124183 shutdown_connections 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.834+0000 7f4d79c76640 1 -- 172.21.15.107:0/1719124183 wait complete. 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 Processor -- start 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 -- start start 2024-04-24T23:43:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d74100060 0x7f4d74199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d741009c0 0x7f4d7419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7410f600 con 0x7f4d74101360 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7410f480 con 0x7f4d74100060 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d79c76640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7410f780 con 0x7f4d741009c0 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.835+0000 7f4d73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36902/0 (socket says 172.21.15.107:36902) 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 -- 172.21.15.107:0/1311068306 learned_addr learned my addr 172.21.15.107:0/1311068306 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d737fe640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d74100060 0x7f4d74199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d72ffd640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d741009c0 0x7f4d7419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 -- 172.21.15.107:0/1311068306 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d741009c0 msgr2=0x7f4d7419a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d741009c0 0x7f4d7419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 -- 172.21.15.107:0/1311068306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d74100060 msgr2=0x7f4d74199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d74100060 0x7f4d74199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 -- 172.21.15.107:0/1311068306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d741a1850 con 0x7f4d74101360 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d73fff640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c005e30 tx=0x7f4d6c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c003c30 con 0x7f4d74101360 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d741a1ae0 con 0x7f4d74101360 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d6c003dd0 con 0x7f4d74101360 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d741a2080 con 0x7f4d74101360 2024-04-24T23:43:12.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.836+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c02fd80 con 0x7f4d74101360 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.837+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7410af00 con 0x7f4d74101360 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.838+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4d6c004000 con 0x7f4d74101360 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.839+0000 7f4d70ff9640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 0x7f4d5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.839+0000 7f4d737fe640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 0x7f4d5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.839+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4d6c0f0810 con 0x7f4d74101360 2024-04-24T23:43:12.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.840+0000 7f4d737fe640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 0x7f4d5c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4d640058c0 tx=0x7f4d6400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:12.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:12.843+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d6c0bd530 con 0x7f4d74101360 2024-04-24T23:43:13.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.020+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d7402d330 con 0x7f4d74101360 2024-04-24T23:43:13.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.021+0000 7f4d70ff9640 1 -- 172.21.15.107:0/1311068306 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f4d6c0bce30 con 0x7f4d74101360 2024-04-24T23:43:13.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:13.022 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:13.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 msgr2=0x7f4d5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 0x7f4d5c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4d640058c0 tx=0x7f4d6400a250 comp rx=0 tx=0).stop 2024-04-24T23:43:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 msgr2=0x7f4d741a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c005e30 tx=0x7f4d6c031ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 shutdown_connections 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4d5c0a2b90 0x7f4d5c0a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d74101360 0x7f4d741a11b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4d741009c0 0x7f4d7419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 --2- 172.21.15.107:0/1311068306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d74100060 0x7f4d74199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 >> 172.21.15.107:0/1311068306 conn(0x7f4d740fbc20 msgr2=0x7f4d740fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 shutdown_connections 2024-04-24T23:43:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.024+0000 7f4d79c76640 1 -- 172.21.15.107:0/1311068306 wait complete. 2024-04-24T23:43:13.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.133+0000 7f654ea88640 1 -- 172.21.15.107:0/1921359282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548102230 msgr2=0x7f6548108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.133+0000 7f654ea88640 1 --2- 172.21.15.107:0/1921359282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548102230 0x7f6548108ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6540009ca0 tx=0x7f654002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.133+0000 7f654ea88640 1 -- 172.21.15.107:0/1921359282 shutdown_connections 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.133+0000 7f654ea88640 1 --2- 172.21.15.107:0/1921359282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548102230 0x7f6548108ac0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 --2- 172.21.15.107:0/1921359282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548101870 0x7f6548101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 --2- 172.21.15.107:0/1921359282 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- 172.21.15.107:0/1921359282 >> 172.21.15.107:0/1921359282 conn(0x7f65480fbbf0 msgr2=0x7f65480fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- 172.21.15.107:0/1921359282 shutdown_connections 2024-04-24T23:43:13.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- 172.21.15.107:0/1921359282 wait complete. 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 Processor -- start 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- start start 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548102230 0x7f654819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f654810d420 con 0x7f6548101870 2024-04-24T23:43:13.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f654810d2a0 con 0x7f6548102230 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f654ea88640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f654810d5a0 con 0x7f6548100680 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f6547fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f6547fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36914/0 (socket says 172.21.15.107:36914) 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.134+0000 7f6547fff640 1 -- 172.21.15.107:0/2182636159 learned_addr learned my addr 172.21.15.107:0/2182636159 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f654d025640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548102230 0x7f654819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f654c824640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 -- 172.21.15.107:0/2182636159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 msgr2=0x7f6548197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 -- 172.21.15.107:0/2182636159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548102230 msgr2=0x7f654819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548102230 0x7f654819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 -- 172.21.15.107:0/2182636159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654819f630 con 0x7f6548101870 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f654c824640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:13.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6547fff640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f653400e9f0 tx=0x7f653400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653400cd80 con 0x7f6548101870 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654819f920 con 0x7f6548101870 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6534010910 con 0x7f6548101870 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65340188f0 con 0x7f6548101870 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.135+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654819fe70 con 0x7f6548101870 2024-04-24T23:43:13.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.136+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65481079a0 con 0x7f6548101870 2024-04-24T23:43:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.137+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6534010ac0 con 0x7f6548101870 2024-04-24T23:43:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.138+0000 7f6545ffb640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 0x7f65380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.138+0000 7f654c824640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 0x7f65380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.138+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65340cf7a0 con 0x7f6548101870 2024-04-24T23:43:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.138+0000 7f654c824640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 0x7f65380a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65481016d0 tx=0x7f653c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:13.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.142+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f653409c4c0 con 0x7f6548101870 2024-04-24T23:43:13.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:13 smithi195 ceph-mon[33019]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:13.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:13 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1311068306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.318+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6548062340 con 0x7f6548101870 2024-04-24T23:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.320+0000 7f6545ffb640 1 -- 172.21.15.107:0/2182636159 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f653409bdc0 con 0x7f6548101870 2024-04-24T23:43:13.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:13.320 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 msgr2=0x7f65380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 0x7f65380a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65481016d0 tx=0x7f653c005f90 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 msgr2=0x7f6548197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f653400e9f0 tx=0x7f653400eeb0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 shutdown_connections 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f65380a2b90 0x7f65380a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6548102230 0x7f654819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6548101870 0x7f6548197f50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 --2- 172.21.15.107:0/2182636159 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6548100680 0x7f6548197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 >> 172.21.15.107:0/2182636159 conn(0x7f65480fbbf0 msgr2=0x7f6548104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 shutdown_connections 2024-04-24T23:43:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.322+0000 7f654ea88640 1 -- 172.21.15.107:0/2182636159 wait complete. 2024-04-24T23:43:13.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:13 smithi107 ceph-mon[29118]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:13.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:13 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1311068306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:13.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T23:43:13.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:13 smithi170 ceph-mon[32714]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:13.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:13 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1311068306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 -- 172.21.15.107:0/1747284190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec101350 msgr2=0x7f18ec1017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 --2- 172.21.15.107:0/1747284190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec101350 0x7f18ec1017d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009ca0 tx=0x7f18e402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 -- 172.21.15.107:0/1747284190 shutdown_connections 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 --2- 172.21.15.107:0/1747284190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec101350 0x7f18ec1017d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 --2- 172.21.15.107:0/1747284190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 0x7f18ec100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 --2- 172.21.15.107:0/1747284190 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec100050 0x7f18ec100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 -- 172.21.15.107:0/1747284190 >> 172.21.15.107:0/1747284190 conn(0x7f18ec0fbbf0 msgr2=0x7f18ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 -- 172.21.15.107:0/1747284190 shutdown_connections 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.434+0000 7f18f2961640 1 -- 172.21.15.107:0/1747284190 wait complete. 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 Processor -- start 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 -- start start 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 0x7f18ec19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 0x7f18ec1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec10f640 con 0x7f18ec100050 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec10f4c0 con 0x7f18ec1009b0 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18f2961640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec10f7c0 con 0x7f18ec101350 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36938/0 (socket says 172.21.15.107:36938) 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18ebfff640 1 -- 172.21.15.107:0/1659496220 learned_addr learned my addr 172.21.15.107:0/1659496220 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.435+0000 7f18eb7fe640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 0x7f18ec19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18f0efe640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 0x7f18ec1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 -- 172.21.15.107:0/1659496220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 msgr2=0x7f18ec1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 0x7f18ec1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 -- 172.21.15.107:0/1659496220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 msgr2=0x7f18ec19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 0x7f18ec19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 -- 172.21.15.107:0/1659496220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ec1a1880 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18f0efe640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 0x7f18ec1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18ebfff640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00c970 tx=0x7f18dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc007bb0 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18dc007d50 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ec1a1b70 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00f450 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.436+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ec1a2070 con 0x7f18ec100050 2024-04-24T23:43:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.437+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18ec10aee0 con 0x7f18ec100050 2024-04-24T23:43:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.438+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f18dc004fe0 con 0x7f18ec100050 2024-04-24T23:43:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.439+0000 7f18e97fa640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 0x7f18d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.439+0000 7f18eb7fe640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 0x7f18d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.439+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f18dc0cf540 con 0x7f18ec100050 2024-04-24T23:43:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.439+0000 7f18eb7fe640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 0x7f18d40a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18e000bbe0 tx=0x7f18e000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:13.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.442+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f18dc014030 con 0x7f18ec100050 2024-04-24T23:43:13.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:13.656+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f18ec02d390 con 0x7f18ec100050 2024-04-24T23:43:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.054+0000 7f18e97fa640 1 -- 172.21.15.107:0/1659496220 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f18dc09c200 con 0x7f18ec100050 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.056+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 msgr2=0x7f18d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.056+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 0x7f18d40a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18e000bbe0 tx=0x7f18e000b390 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.056+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 msgr2=0x7f18ec199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.056+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00c970 tx=0x7f18dc00ce30 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 shutdown_connections 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18d40a2b90 0x7f18d40a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18ec101350 0x7f18ec1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18ec1009b0 0x7f18ec19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 --2- 172.21.15.107:0/1659496220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18ec100050 0x7f18ec199c70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 >> 172.21.15.107:0/1659496220 conn(0x7f18ec0fbbf0 msgr2=0x7f18ec0fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 shutdown_connections 2024-04-24T23:43:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.057+0000 7f18f2961640 1 -- 172.21.15.107:0/1659496220 wait complete. 2024-04-24T23:43:14.068 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-24T23:43:14.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-24T23:43:14.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 -- 172.21.15.107:0/1896064592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 msgr2=0x7f8d0c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:14.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 --2- 172.21.15.107:0/1896064592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c101bc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009900 tx=0x7f8cfc02a410 comp rx=0 tx=0).stop 2024-04-24T23:43:14.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 -- 172.21.15.107:0/1896064592 shutdown_connections 2024-04-24T23:43:14.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 --2- 172.21.15.107:0/1896064592 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 0x7f8d0c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 --2- 172.21.15.107:0/1896064592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 --2- 172.21.15.107:0/1896064592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c101bc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 -- 172.21.15.107:0/1896064592 >> 172.21.15.107:0/1896064592 conn(0x7f8d0c0fcd10 msgr2=0x7f8d0c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 -- 172.21.15.107:0/1896064592 shutdown_connections 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.170+0000 7f8d10aae640 1 -- 172.21.15.107:0/1896064592 wait complete. 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 Processor -- start 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 -- start start 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 0x7f8d0c19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d4f0 con 0x7f8d0c1017a0 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d370 con 0x7f8d0c102990 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d10aae640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d670 con 0x7f8d0c103350 2024-04-24T23:43:14.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36952/0 (socket says 172.21.15.107:36952) 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0a59c640 1 -- 172.21.15.107:0/3069921467 learned_addr learned my addr 172.21.15.107:0/3069921467 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d01d9b640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0ad9d640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 0x7f8d0c19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0a59c640 1 -- 172.21.15.107:0/3069921467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 msgr2=0x7f8d0c19f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.171+0000 7f8d0a59c640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 0x7f8d0c19f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d0a59c640 1 -- 172.21.15.107:0/3069921467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 msgr2=0x7f8d0c198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d0a59c640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d0a59c640 1 -- 172.21.15.107:0/3069921467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c19f860 con 0x7f8d0c1017a0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d01d9b640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d0a59c640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc031610 tx=0x7f8cfc0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc03f030 con 0x7f8d0c1017a0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cfc003ce0 con 0x7f8d0c1017a0 2024-04-24T23:43:14.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c19faf0 con 0x7f8d0c1017a0 2024-04-24T23:43:14.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc035070 con 0x7f8d0c1017a0 2024-04-24T23:43:14.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.172+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c19ffe0 con 0x7f8d0c1017a0 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.173+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0c108be0 con 0x7f8d0c1017a0 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.174+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8cfc042030 con 0x7f8d0c1017a0 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.174+0000 7f8d03fff640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 0x7f8cf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.175+0000 7f8d01d9b640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 0x7f8cf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.175+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8cfc0efa70 con 0x7f8d0c1017a0 2024-04-24T23:43:14.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.175+0000 7f8d01d9b640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 0x7f8cf80a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009930 tx=0x7f8cf4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.179+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cfc0bc790 con 0x7f8d0c1017a0 2024-04-24T23:43:14.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:14 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2182636159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:14.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:14 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T23:43:14.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:14 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2182636159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:14.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:14 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T23:43:14.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2182636159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:14.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:14 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T23:43:14.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:14.394+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f8d0c062340 con 0x7f8d0c1017a0 2024-04-24T23:43:15.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.056+0000 7f8d03fff640 1 -- 172.21.15.107:0/3069921467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f8cfc0bc090 con 0x7f8d0c1017a0 2024-04-24T23:43:15.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.058+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 msgr2=0x7f8cf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 0x7f8cf80a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009930 tx=0x7f8cf4008040 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 msgr2=0x7f8d0c197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc031610 tx=0x7f8cfc0043b0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 shutdown_connections 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8cf80a2b90 0x7f8cf80a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d0c103350 0x7f8d0c19f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c102990 0x7f8d0c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 --2- 172.21.15.107:0/3069921467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c1017a0 0x7f8d0c197c50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 >> 172.21.15.107:0/3069921467 conn(0x7f8d0c0fcd10 msgr2=0x7f8d0c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 shutdown_connections 2024-04-24T23:43:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.059+0000 7f8d10aae640 1 -- 172.21.15.107:0/3069921467 wait complete. 2024-04-24T23:43:15.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 -- 172.21.15.107:0/1919349078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150102250 msgr2=0x7f2150108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 --2- 172.21.15.107:0/1919349078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150102250 0x7f2150108ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2148009ca0 tx=0x7f214802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 -- 172.21.15.107:0/1919349078 shutdown_connections 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 --2- 172.21.15.107:0/1919349078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150102250 0x7f2150108ae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 --2- 172.21.15.107:0/1919349078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150101890 0x7f2150101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 --2- 172.21.15.107:0/1919349078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 -- 172.21.15.107:0/1919349078 >> 172.21.15.107:0/1919349078 conn(0x7f21500fbc10 msgr2=0x7f21500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 -- 172.21.15.107:0/1919349078 shutdown_connections 2024-04-24T23:43:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 -- 172.21.15.107:0/1919349078 wait complete. 2024-04-24T23:43:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.182+0000 7f2156d16640 1 Processor -- start 2024-04-24T23:43:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 -- start start 2024-04-24T23:43:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150102250 0x7f215019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f215010c370 con 0x7f2150101890 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f215010c1f0 con 0x7f2150102250 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f215010c4f0 con 0x7f21501006a0 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36968/0 (socket says 172.21.15.107:36968) 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 -- 172.21.15.107:0/1950893234 learned_addr learned my addr 172.21.15.107:0/1950893234 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2154ab2640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f21552b3640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150102250 0x7f215019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 -- 172.21.15.107:0/1950893234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 msgr2=0x7f2150197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 -- 172.21.15.107:0/1950893234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150102250 msgr2=0x7f215019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150102250 0x7f215019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 -- 172.21.15.107:0/1950893234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215019f660 con 0x7f2150101890 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2154ab2640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214ffff640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f213c00c920 tx=0x7f213c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c007c70 con 0x7f2150101890 2024-04-24T23:43:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f213c007e10 con 0x7f2150101890 2024-04-24T23:43:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c018890 con 0x7f2150101890 2024-04-24T23:43:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.183+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215019f950 con 0x7f2150101890 2024-04-24T23:43:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.184+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215019fe50 con 0x7f2150101890 2024-04-24T23:43:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.185+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21501a00e0 con 0x7f2150101890 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.185+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f213c018a70 con 0x7f2150101890 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.186+0000 7f214dffb640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 0x7f21400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.186+0000 7f2154ab2640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 0x7f21400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.186+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f213c0cf450 con 0x7f2150101890 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.186+0000 7f2154ab2640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 0x7f21400a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21440058c0 tx=0x7f2144005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.190+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f213c014030 con 0x7f2150101890 2024-04-24T23:43:15.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:15 smithi195 ceph-mon[33019]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:15.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:15 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T23:43:15.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:15 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:15.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:15 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-24T23:43:15.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:15 smithi170 ceph-mon[32714]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:15.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:15 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T23:43:15.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:15 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:15.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:15 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-24T23:43:15.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:15 smithi107 ceph-mon[29118]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-24T23:43:15.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:15 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1659496220' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T23:43:15.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:15 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:15.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:15 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-24T23:43:15.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:15.402+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f21501a0310 con 0x7f2150101890 2024-04-24T23:43:16.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.066+0000 7f214dffb640 1 -- 172.21.15.107:0/1950893234 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f213c09c170 con 0x7f2150101890 2024-04-24T23:43:16.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.068+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 msgr2=0x7f21400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.068+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 0x7f21400a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21440058c0 tx=0x7f2144005850 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.068+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 msgr2=0x7f2150197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.068+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f213c00c920 tx=0x7f213c00cde0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 shutdown_connections 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f21400a2c10 0x7f21400a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2150102250 0x7f215019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2150101890 0x7f2150197f80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 --2- 172.21.15.107:0/1950893234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f21501006a0 0x7f2150197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 >> 172.21.15.107:0/1950893234 conn(0x7f21500fbc10 msgr2=0x7f21500fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 shutdown_connections 2024-04-24T23:43:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.069+0000 7f2156d16640 1 -- 172.21.15.107:0/1950893234 wait complete. 2024-04-24T23:43:16.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-24T23:43:16.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 -- 172.21.15.107:0/1842151408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74104b30 msgr2=0x7fda74106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:16.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 --2- 172.21.15.107:0/1842151408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74104b30 0x7fda74106f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fda68009900 tx=0x7fda6802a410 comp rx=0 tx=0).stop 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 -- 172.21.15.107:0/1842151408 shutdown_connections 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 --2- 172.21.15.107:0/1842151408 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 0x7fda741098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 --2- 172.21.15.107:0/1842151408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74104b30 0x7fda74106f20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 --2- 172.21.15.107:0/1842151408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74102200 0x7fda741045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 -- 172.21.15.107:0/1842151408 >> 172.21.15.107:0/1842151408 conn(0x7fda740fbbf0 msgr2=0x7fda740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 -- 172.21.15.107:0/1842151408 shutdown_connections 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.186+0000 7fda79329640 1 -- 172.21.15.107:0/1842151408 wait complete. 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 Processor -- start 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 -- start start 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74104b30 0x7fda741001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 0x7fda741006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda7410d450 con 0x7fda74102200 2024-04-24T23:43:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fda7410d2d0 con 0x7fda74104b30 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda79329640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fda7410d5d0 con 0x7fda74107460 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36978/0 (socket says 172.21.15.107:36978) 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda72ffd640 1 -- 172.21.15.107:0/3950928131 learned_addr learned my addr 172.21.15.107:0/3950928131 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda727fc640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74104b30 0x7fda741001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.187+0000 7fda737fe640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 0x7fda741006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 -- 172.21.15.107:0/3950928131 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 msgr2=0x7fda741006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 0x7fda741006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 -- 172.21.15.107:0/3950928131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74104b30 msgr2=0x7fda741001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74104b30 0x7fda741001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 -- 172.21.15.107:0/3950928131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda74100f80 con 0x7fda74102200 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda72ffd640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fda6400e9f0 tx=0x7fda6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:16.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6400cd80 con 0x7fda74102200 2024-04-24T23:43:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda64010910 con 0x7fda74102200 2024-04-24T23:43:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda741a1c40 con 0x7fda74102200 2024-04-24T23:43:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64018990 con 0x7fda74102200 2024-04-24T23:43:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.188+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda741a2160 con 0x7fda74102200 2024-04-24T23:43:16.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.189+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda740fd6e0 con 0x7fda74102200 2024-04-24T23:43:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.190+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fda64010ac0 con 0x7fda74102200 2024-04-24T23:43:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.190+0000 7fda5bfff640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 0x7fda5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.191+0000 7fda727fc640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 0x7fda5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.191+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fda640cf630 con 0x7fda74102200 2024-04-24T23:43:16.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.191+0000 7fda727fc640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 0x7fda5c0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda6802aa00 tx=0x7fda680315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.195+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda6409c350 con 0x7fda74102200 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-24T23:43:16.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-24T23:43:16.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T23:43:16.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T23:43:16.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-24T23:43:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T23:43:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-24T23:43:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-24T23:43:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T23:43:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:16.410+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fda74062370 con 0x7fda74102200 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3069921467' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-24T23:43:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T23:43:17.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.074+0000 7fda5bfff640 1 -- 172.21.15.107:0/3950928131 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fda6409bc50 con 0x7fda74102200 2024-04-24T23:43:17.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 msgr2=0x7fda5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 0x7fda5c0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda6802aa00 tx=0x7fda680315b0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 msgr2=0x7fda74101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fda6400e9f0 tx=0x7fda6400eeb0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 shutdown_connections 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda5c0a2b90 0x7fda5c0a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda74107460 0x7fda741006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda74104b30 0x7fda741001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 --2- 172.21.15.107:0/3950928131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda74102200 0x7fda74101b70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 >> 172.21.15.107:0/3950928131 conn(0x7fda740fbbf0 msgr2=0x7fda74102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 shutdown_connections 2024-04-24T23:43:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.077+0000 7fda79329640 1 -- 172.21.15.107:0/3950928131 wait complete. 2024-04-24T23:43:17.094 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-24T23:43:17.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-24T23:43:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/2304583717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4102980 msgr2=0x7fc1c4102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/2304583717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4102980 0x7fc1c4102e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009900 tx=0x7fc1b802a410 comp rx=0 tx=0).stop 2024-04-24T23:43:17.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/2304583717 shutdown_connections 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/2304583717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4103340 0x7fc1c4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/2304583717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4102980 0x7fc1c4102e00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/2304583717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 0x7fc1c4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/2304583717 >> 172.21.15.107:0/2304583717 conn(0x7fc1c40fcd20 msgr2=0x7fc1c40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/2304583717 shutdown_connections 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.201+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/2304583717 wait complete. 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 Processor -- start 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 -- start start 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 0x7fc1c4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4102980 0x7fc1c41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c410d480 con 0x7fc1c4103340 2024-04-24T23:43:17.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c410d300 con 0x7fc1c4102980 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1ca9ab640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c410d600 con 0x7fc1c4101790 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1c8f48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1c8f48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42460/0 (socket says 172.21.15.107:42460) 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1c8f48640 1 -- 172.21.15.107:0/367448614 learned_addr learned my addr 172.21.15.107:0/367448614 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1c37fe640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4102980 0x7fc1c41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.202+0000 7fc1c3fff640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 0x7fc1c4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 -- 172.21.15.107:0/367448614 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 msgr2=0x7fc1c4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 0x7fc1c4197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 -- 172.21.15.107:0/367448614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4102980 msgr2=0x7fc1c41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4102980 0x7fc1c41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 -- 172.21.15.107:0/367448614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c419f7a0 con 0x7fc1c4103340 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c8f48640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009fd0 tx=0x7fc1bc00def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:17.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc017070 con 0x7fc1c4103340 2024-04-24T23:43:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c419fa90 con 0x7fc1c4103340 2024-04-24T23:43:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc003e50 con 0x7fc1c4103340 2024-04-24T23:43:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc010460 con 0x7fc1c4103340 2024-04-24T23:43:17.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.203+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c419ff60 con 0x7fc1c4103340 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.204+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c4108b80 con 0x7fc1c4103340 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.205+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc1bc003970 con 0x7fc1c4103340 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.206+0000 7fc1c17fa640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 0x7fc1ac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.206+0000 7fc1c3fff640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 0x7fc1ac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.206+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1bc0cef40 con 0x7fc1c4103340 2024-04-24T23:43:17.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.206+0000 7fc1c3fff640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 0x7fc1ac0a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc1c41027e0 tx=0x7fc1b4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:17.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.211+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc1bc09bc60 con 0x7fc1c4103340 2024-04-24T23:43:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-24T23:43:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T23:43:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:active 2024-04-24T23:43:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T23:43:17.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-24T23:43:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-24T23:43:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-24T23:43:17.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:17 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T23:43:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-24T23:43:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:active 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-24T23:43:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T23:43:17.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:17.410+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc1c4061b80 con 0x7fc1ac0a2b90 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1950893234' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] up:active 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-24T23:43:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-24T23:43:17.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-24T23:43:17.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:17 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T23:43:17.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:43:17.415+0000 7f44ff21a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T23:43:17.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:43:17.415+0000 7f44ff21a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T23:43:17.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:43:17.415+0000 7f44ff21a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T23:43:17.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:43:17.415+0000 7f44ff21a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T23:43:17.878 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:43:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:43:17.415+0000 7f44ff21a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T23:43:18.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.108+0000 7fc1c17fa640 1 -- 172.21.15.107:0/367448614 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc1c4061b80 con 0x7fc1ac0a2b90 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 msgr2=0x7fc1ac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 0x7fc1ac0a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc1c41027e0 tx=0x7fc1b4009110 comp rx=0 tx=0).stop 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 msgr2=0x7fc1c419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009fd0 tx=0x7fc1bc00def0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 shutdown_connections 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc1ac0a2b90 0x7fc1ac0a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c4103340 0x7fc1c419f130 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c4102980 0x7fc1c41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 --2- 172.21.15.107:0/367448614 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc1c4101790 0x7fc1c4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 >> 172.21.15.107:0/367448614 conn(0x7fc1c40fcd20 msgr2=0x7fc1c4105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 shutdown_connections 2024-04-24T23:43:18.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.110+0000 7fc1ca9ab640 1 -- 172.21.15.107:0/367448614 wait complete. 2024-04-24T23:43:18.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-24T23:43:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- 172.21.15.107:0/2148309949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c101870 msgr2=0x7f3e1c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/2148309949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c101870 0x7f3e1c101cf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009900 tx=0x7f3e0402a410 comp rx=0 tx=0).stop 2024-04-24T23:43:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- 172.21.15.107:0/2148309949 shutdown_connections 2024-04-24T23:43:18.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/2148309949 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c102230 0x7f3e1c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/2148309949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c101870 0x7f3e1c101cf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/2148309949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c100680 0x7f3e1c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- 172.21.15.107:0/2148309949 >> 172.21.15.107:0/2148309949 conn(0x7f3e1c0fbbf0 msgr2=0x7f3e1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- 172.21.15.107:0/2148309949 shutdown_connections 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- 172.21.15.107:0/2148309949 wait complete. 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 Processor -- start 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.226+0000 7f3e21ee1640 1 -- start start 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c100680 0x7f3e1c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c101870 0x7f3e1c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c10c350 con 0x7f3e1c102230 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c10c1d0 con 0x7f3e1c101870 2024-04-24T23:43:18.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c10c4d0 con 0x7f3e1c100680 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42494/0 (socket says 172.21.15.107:42494) 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 -- 172.21.15.107:0/3506598527 learned_addr learned my addr 172.21.15.107:0/3506598527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e13fff640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c101870 0x7f3e1c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1b7fe640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c100680 0x7f3e1c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 -- 172.21.15.107:0/3506598527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c100680 msgr2=0x7f3e1c069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c100680 0x7f3e1c069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 -- 172.21.15.107:0/3506598527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c101870 msgr2=0x7f3e1c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c101870 0x7f3e1c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 -- 172.21.15.107:0/3506598527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c19f6c0 con 0x7f3e1c102230 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e1bfff640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400eb10 tx=0x7f3e1400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14004b20 con 0x7f3e1c102230 2024-04-24T23:43:18.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14004cc0 con 0x7f3e1c102230 2024-04-24T23:43:18.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c19f9b0 con 0x7f3e1c102230 2024-04-24T23:43:18.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14010760 con 0x7f3e1c102230 2024-04-24T23:43:18.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.227+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c19fed0 con 0x7f3e1c102230 2024-04-24T23:43:18.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.228+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c1079a0 con 0x7f3e1c102230 2024-04-24T23:43:18.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.229+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3e14005000 con 0x7f3e1c102230 2024-04-24T23:43:18.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.230+0000 7f3e197fa640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 0x7f3e080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.230+0000 7f3e1b7fe640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 0x7f3e080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.230+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3e140cfd50 con 0x7f3e1c102230 2024-04-24T23:43:18.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.230+0000 7f3e1b7fe640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 0x7f3e080a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c1016d0 tx=0x7f3e0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.233+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e1409ca70 con 0x7f3e1c102230 2024-04-24T23:43:18.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T23:43:18.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:active 2024-04-24T23:43:18.337 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-24T23:43:18.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-24T23:43:18.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-24T23:43:18.338 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:18 smithi170 ceph-mon[32714]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-24T23:43:18.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T23:43:18.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:active 2024-04-24T23:43:18.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-24T23:43:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-24T23:43:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-24T23:43:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:18 smithi107 ceph-mon[29118]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-24T23:43:18.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.437+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3e1c061b80 con 0x7f3e080a2b90 2024-04-24T23:43:18.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.467+0000 7f3e197fa640 1 -- 172.21.15.107:0/3506598527 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3e1c061b80 con 0x7f3e080a2b90 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 msgr2=0x7f3e080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 0x7f3e080a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c1016d0 tx=0x7f3e0c005f90 comp rx=0 tx=0).stop 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 msgr2=0x7f3e1c19ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400eb10 tx=0x7f3e1400c2d0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 shutdown_connections 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3e080a2b90 0x7f3e080a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1c102230 0x7f3e1c19ee00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c101870 0x7f3e1c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 --2- 172.21.15.107:0/3506598527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3e1c100680 0x7f3e1c069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 >> 172.21.15.107:0/3506598527 conn(0x7f3e1c0fbbf0 msgr2=0x7f3e1c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 shutdown_connections 2024-04-24T23:43:18.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.469+0000 7f3e21ee1640 1 -- 172.21.15.107:0/3506598527 wait complete. 2024-04-24T23:43:18.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3950928131' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] up:active 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-24T23:43:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:18 smithi195 ceph-mon[33019]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.583+0000 7f08b0110640 1 -- 172.21.15.107:0/1699485457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8100680 msgr2=0x7f08a8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.583+0000 7f08b0110640 1 --2- 172.21.15.107:0/1699485457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8100680 0x7f08a8100aa0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0898009900 tx=0x7f089802a410 comp rx=0 tx=0).stop 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- 172.21.15.107:0/1699485457 shutdown_connections 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- 172.21.15.107:0/1699485457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8102230 0x7f08a8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- 172.21.15.107:0/1699485457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8101870 0x7f08a8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- 172.21.15.107:0/1699485457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8100680 0x7f08a8100aa0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- 172.21.15.107:0/1699485457 >> 172.21.15.107:0/1699485457 conn(0x7f08a80fbbf0 msgr2=0x7f08a80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- 172.21.15.107:0/1699485457 shutdown_connections 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- 172.21.15.107:0/1699485457 wait complete. 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 Processor -- start 2024-04-24T23:43:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- start start 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 0x7f08a8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 0x7f08a819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810c340 con 0x7f08a8101870 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810c1c0 con 0x7f08a8100680 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08b0110640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810c4c0 con 0x7f08a8102230 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08ad6ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08ad6ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42512/0 (socket says 172.21.15.107:42512) 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.584+0000 7f08ad6ab640 1 -- 172.21.15.107:0/1650612535 learned_addr learned my addr 172.21.15.107:0/1650612535 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08adeac640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 0x7f08a8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 -- 172.21.15.107:0/1650612535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 msgr2=0x7f08a819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ae6ad640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 0x7f08a819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 0x7f08a819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 -- 172.21.15.107:0/1650612535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 msgr2=0x7f08a8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 0x7f08a8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 -- 172.21.15.107:0/1650612535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a819f730 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ae6ad640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 0x7f08a819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08adeac640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 0x7f08a8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08ad6ab640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f08a400d770 tx=0x7f08a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4017070 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a4005500 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a819fa20 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4004e00 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.585+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a819ff20 con 0x7f08a8101870 2024-04-24T23:43:18.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.586+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a81a0150 con 0x7f08a8101870 2024-04-24T23:43:18.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.587+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f08a4004fa0 con 0x7f08a8101870 2024-04-24T23:43:18.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.588+0000 7f089effd640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 0x7f08940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.588+0000 7f08adeac640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 0x7f08940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.588+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f08a40ced00 con 0x7f08a8101870 2024-04-24T23:43:18.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.589+0000 7f08adeac640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 0x7f08940a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f089802aa00 tx=0x7f08980315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.593+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08a409ba20 con 0x7f08a8101870 2024-04-24T23:43:18.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.794+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f08a80fddc0 con 0x7f08940a2b90 2024-04-24T23:43:18.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.799+0000 7f089effd640 1 -- 172.21.15.107:0/1650612535 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f08a80fddc0 con 0x7f08940a2b90 2024-04-24T23:43:18.799 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_0/67c35d55-bad6-43b5-8c05-a98bf9118690 2024-04-24T23:43:18.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 msgr2=0x7f08940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 0x7f08940a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f089802aa00 tx=0x7f08980315b0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 msgr2=0x7f08a8198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f08a400d770 tx=0x7f08a400dc30 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 shutdown_connections 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08940a2b90 0x7f08940a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08a8102230 0x7f08a819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a8101870 0x7f08a8198050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.801+0000 7f08b0110640 1 --2- 172.21.15.107:0/1650612535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08a8100680 0x7f08a8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.802+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 >> 172.21.15.107:0/1650612535 conn(0x7f08a80fbbf0 msgr2=0x7f08a80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.802+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 shutdown_connections 2024-04-24T23:43:18.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.802+0000 7f08b0110640 1 -- 172.21.15.107:0/1650612535 wait complete. 2024-04-24T23:43:18.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- 172.21.15.107:0/1322350297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 msgr2=0x7f4c90100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1322350297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90100aa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009ca0 tx=0x7f4c8802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- 172.21.15.107:0/1322350297 shutdown_connections 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1322350297 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90102230 0x7f4c90108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1322350297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90101870 0x7f4c90101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1322350297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90100aa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- 172.21.15.107:0/1322350297 >> 172.21.15.107:0/1322350297 conn(0x7f4c900fbbf0 msgr2=0x7f4c900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- 172.21.15.107:0/1322350297 shutdown_connections 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- 172.21.15.107:0/1322350297 wait complete. 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 Processor -- start 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.915+0000 7f4c978a5640 1 -- start start 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 0x7f4c90102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90102230 0x7f4c90103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010c300 con 0x7f4c90100680 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010c180 con 0x7f4c90102230 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010c480 con 0x7f4c90101870 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42520/0 (socket says 172.21.15.107:42520) 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 -- 172.21.15.107:0/1376340257 learned_addr learned my addr 172.21.15.107:0/1376340257 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95e42640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90102230 0x7f4c90103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c94e40640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 0x7f4c90102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 -- 172.21.15.107:0/1376340257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 msgr2=0x7f4c90102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 0x7f4c90102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 -- 172.21.15.107:0/1376340257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90102230 msgr2=0x7f4c90103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90102230 0x7f4c90103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 -- 172.21.15.107:0/1376340257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c90103a20 con 0x7f4c90100680 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c94e40640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 0x7f4c90102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c95641640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009bf0 tx=0x7f4c88031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8802fae0 con 0x7f4c90100680 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c901a1c20 con 0x7f4c90100680 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8802fc80 con 0x7f4c90100680 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c880467e0 con 0x7f4c90100680 2024-04-24T23:43:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.916+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c901a2170 con 0x7f4c90100680 2024-04-24T23:43:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.917+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c901a2340 con 0x7f4c90100680 2024-04-24T23:43:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.918+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4c88030b30 con 0x7f4c90100680 2024-04-24T23:43:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.919+0000 7f4c827fc640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 0x7f4c7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.919+0000 7f4c94e40640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 0x7f4c7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.919+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4c880f0a70 con 0x7f4c90100680 2024-04-24T23:43:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.919+0000 7f4c94e40640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 0x7f4c7c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c90104380 tx=0x7f4c84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:18.922+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4c880bd820 con 0x7f4c90100680 2024-04-24T23:43:19.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.128+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4c900fde40 con 0x7f4c7c0a2b90 2024-04-24T23:43:19.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.153+0000 7f4c827fc640 1 -- 172.21.15.107:0/1376340257 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4c900fde40 con 0x7f4c7c0a2b90 2024-04-24T23:43:19.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 msgr2=0x7f4c7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 0x7f4c7c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c90104380 tx=0x7f4c84005f90 comp rx=0 tx=0).stop 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 msgr2=0x7f4c90104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009bf0 tx=0x7f4c88031ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 shutdown_connections 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4c7c0a2b90 0x7f4c7c0a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90102230 0x7f4c90103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4c90101870 0x7f4c90102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 --2- 172.21.15.107:0/1376340257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c90100680 0x7f4c90104610 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.155+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 >> 172.21.15.107:0/1376340257 conn(0x7f4c900fbbf0 msgr2=0x7f4c900fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.156+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 shutdown_connections 2024-04-24T23:43:19.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.156+0000 7f4c978a5640 1 -- 172.21.15.107:0/1376340257 wait complete. 2024-04-24T23:43:19.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-24T23:43:19.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 -- 172.21.15.107:0/943913525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0101890 msgr2=0x7f40f0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/943913525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0101890 0x7f40f0101d10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009ca0 tx=0x7f40e002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 -- 172.21.15.107:0/943913525 shutdown_connections 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/943913525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0102250 0x7f40f0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/943913525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0101890 0x7f40f0101d10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/943913525 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 -- 172.21.15.107:0/943913525 >> 172.21.15.107:0/943913525 conn(0x7f40f00fbc30 msgr2=0x7f40f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 -- 172.21.15.107:0/943913525 shutdown_connections 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.269+0000 7f40f6b3b640 1 -- 172.21.15.107:0/943913525 wait complete. 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 Processor -- start 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 -- start start 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0101890 0x7f40f0196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40f010c300 con 0x7f40f0102250 2024-04-24T23:43:19.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f40f010c180 con 0x7f40f0101890 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f6b3b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f40f010c480 con 0x7f40f01006a0 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42538/0 (socket says 172.21.15.107:42538) 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 -- 172.21.15.107:0/1965073360 learned_addr learned my addr 172.21.15.107:0/1965073360 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40effff640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0101890 0x7f40f0196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f48d7640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 -- 172.21.15.107:0/1965073360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 msgr2=0x7f40f0196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 -- 172.21.15.107:0/1965073360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0101890 msgr2=0x7f40f0196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0101890 0x7f40f0196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 -- 172.21.15.107:0/1965073360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f019e560 con 0x7f40f0102250 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f48d7640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.270+0000 7f40f50d8640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f40d800e970 tx=0x7f40d800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.271+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d800ccc0 con 0x7f40f0102250 2024-04-24T23:43:19.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.271+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40d800ce60 con 0x7f40f0102250 2024-04-24T23:43:19.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.271+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f019e850 con 0x7f40f0102250 2024-04-24T23:43:19.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.271+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8010490 con 0x7f40f0102250 2024-04-24T23:43:19.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.271+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f019ed50 con 0x7f40f0102250 2024-04-24T23:43:19.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.272+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f0107a10 con 0x7f40f0102250 2024-04-24T23:43:19.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.273+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f40d8005020 con 0x7f40f0102250 2024-04-24T23:43:19.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.273+0000 7f40edffb640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 0x7f40dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.273+0000 7f40f48d7640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 0x7f40dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.273+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f40d80cf780 con 0x7f40f0102250 2024-04-24T23:43:19.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.274+0000 7f40f48d7640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 0x7f40dc0a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f40f01016f0 tx=0x7f40e800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.278+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f40d809c4a0 con 0x7f40f0102250 2024-04-24T23:43:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-24T23:43:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:active 2024-04-24T23:43:19.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-24T23:43:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T23:43:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: Cluster is now healthy 2024-04-24T23:43:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-24T23:43:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-24T23:43:19.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:19 smithi170 ceph-mon[32714]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-24T23:43:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:active 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: Cluster is now healthy 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-24T23:43:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:19 smithi107 ceph-mon[29118]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-24T23:43:19.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.485+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f40f0061b80 con 0x7f40dc0a2b90 2024-04-24T23:43:19.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.491+0000 7f40edffb640 1 -- 172.21.15.107:0/1965073360 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f40f0061b80 con 0x7f40dc0a2b90 2024-04-24T23:43:19.490 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_1/14a26539-10ae-41c8-bb82-2677aaedd4e9 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 msgr2=0x7f40dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 0x7f40dc0a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f40f01016f0 tx=0x7f40e800a5e0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 msgr2=0x7f40f019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f40d800e970 tx=0x7f40d800ed70 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 shutdown_connections 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f40dc0a2b90 0x7f40dc0a5050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40f0102250 0x7f40f019def0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40f0101890 0x7f40f0196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 --2- 172.21.15.107:0/1965073360 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f40f01006a0 0x7f40f0196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 >> 172.21.15.107:0/1965073360 conn(0x7f40f00fbc30 msgr2=0x7f40f0104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 shutdown_connections 2024-04-24T23:43:19.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.493+0000 7f40f6b3b640 1 -- 172.21.15.107:0/1965073360 wait complete. 2024-04-24T23:43:19.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] up:active 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: Cluster is now healthy 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-24T23:43:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:19 smithi195 ceph-mon[33019]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-24T23:43:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1977362683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0100680 msgr2=0x7fa7e0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1977362683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0100680 0x7fa7e0100aa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009900 tx=0x7fa7d002a410 comp rx=0 tx=0).stop 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1977362683 shutdown_connections 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1977362683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0102230 0x7fa7e0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1977362683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0101870 0x7fa7e0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1977362683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0100680 0x7fa7e0100aa0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1977362683 >> 172.21.15.107:0/1977362683 conn(0x7fa7e00fbbf0 msgr2=0x7fa7e00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1977362683 shutdown_connections 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.602+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1977362683 wait complete. 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 Processor -- start 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 -- start start 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0100680 0x7fa7e0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0101870 0x7fa7e0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e010c2d0 con 0x7fa7e0102230 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e010c150 con 0x7fa7e0101870 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e84bd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e010c450 con 0x7fa7e0100680 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e6a5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e6a5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42542/0 (socket says 172.21.15.107:42542) 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e6a5a640 1 -- 172.21.15.107:0/1874735645 learned_addr learned my addr 172.21.15.107:0/1874735645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e5a58640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0101870 0x7fa7e0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.603+0000 7fa7e6259640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0100680 0x7fa7e0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 -- 172.21.15.107:0/1874735645 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0100680 msgr2=0x7fa7e0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0100680 0x7fa7e0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 -- 172.21.15.107:0/1874735645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0101870 msgr2=0x7fa7e0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0101870 0x7fa7e0197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 -- 172.21.15.107:0/1874735645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e019f660 con 0x7fa7e0102230 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e6a5a640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800eb10 tx=0x7fa7d800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800ce20 con 0x7fa7e0102230 2024-04-24T23:43:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e019f950 con 0x7fa7e0102230 2024-04-24T23:43:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d8005020 con 0x7fa7e0102230 2024-04-24T23:43:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8010690 con 0x7fa7e0102230 2024-04-24T23:43:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.604+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e019fe50 con 0x7fa7e0102230 2024-04-24T23:43:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.605+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e01a0140 con 0x7fa7e0102230 2024-04-24T23:43:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.606+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa7d8005540 con 0x7fa7e0102230 2024-04-24T23:43:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.606+0000 7fa7d77fe640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.606+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa7d80cf8c0 con 0x7fa7e0102230 2024-04-24T23:43:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.606+0000 7fa7e6259640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.607+0000 7fa7e6259640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0006010 tx=0x7fa7d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.611+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa7d809c5e0 con 0x7fa7e0102230 2024-04-24T23:43:19.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.822+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e01a0310 con 0x7fa7e0102230 2024-04-24T23:43:19.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.823+0000 7fa7d77fe640 1 -- 172.21.15.107:0/1874735645 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7fa7d809bee0 con 0x7fa7e0102230 2024-04-24T23:43:19.823 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:19.823 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:43:19.823 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 msgr2=0x7fa7cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0006010 tx=0x7fa7d0000bc0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 msgr2=0x7fa7e019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800eb10 tx=0x7fa7d800ef10 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 shutdown_connections 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7cc0a2b90 0x7fa7cc0a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7e0102230 0x7fa7e019eff0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e0101870 0x7fa7e0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 --2- 172.21.15.107:0/1874735645 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7e0100680 0x7fa7e0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 >> 172.21.15.107:0/1874735645 conn(0x7fa7e00fbbf0 msgr2=0x7fa7e00688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 shutdown_connections 2024-04-24T23:43:19.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.825+0000 7fa7e84bd640 1 -- 172.21.15.107:0/1874735645 wait complete. 2024-04-24T23:43:19.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- 172.21.15.107:0/1421736727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 msgr2=0x7f06c4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1421736727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c4108ae0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009ca0 tx=0x7f06bc02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- 172.21.15.107:0/1421736727 shutdown_connections 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1421736727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c4108ae0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1421736727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 0x7f06c4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1421736727 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- 172.21.15.107:0/1421736727 >> 172.21.15.107:0/1421736727 conn(0x7f06c40fbc30 msgr2=0x7f06c40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- 172.21.15.107:0/1421736727 shutdown_connections 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- 172.21.15.107:0/1421736727 wait complete. 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 Processor -- start 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.934+0000 7f06cb81c640 1 -- start start 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c406a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 0x7f06c419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06c410c390 con 0x7f06c4102250 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06c410c210 con 0x7f06c4101890 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f06c410c510 con 0x7f06c41006a0 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42562/0 (socket says 172.21.15.107:42562) 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 -- 172.21.15.107:0/1988015121 learned_addr learned my addr 172.21.15.107:0/1988015121 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c8db7640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 0x7f06c419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 -- 172.21.15.107:0/1988015121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 msgr2=0x7f06c406a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c95b8640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c406a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c406a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 -- 172.21.15.107:0/1988015121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 msgr2=0x7f06c419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 0x7f06c419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 -- 172.21.15.107:0/1988015121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c419f710 con 0x7f06c4102250 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c95b8640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c406a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06c9db9640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06bc031ae0 tx=0x7f06bc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc003c30 con 0x7f06c4102250 2024-04-24T23:43:19.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c419f9a0 con 0x7f06c4102250 2024-04-24T23:43:19.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06bc003dd0 con 0x7f06c4102250 2024-04-24T23:43:19.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc02fcb0 con 0x7f06c4102250 2024-04-24T23:43:19.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.935+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c419ff70 con 0x7f06c4102250 2024-04-24T23:43:19.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.936+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c4107a10 con 0x7f06c4102250 2024-04-24T23:43:19.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.937+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f06bc004050 con 0x7f06c4102250 2024-04-24T23:43:19.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.938+0000 7f06ba7fc640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 0x7f06b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:19.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.938+0000 7f06c95b8640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 0x7f06b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:19.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.938+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f06bc0f0750 con 0x7f06c4102250 2024-04-24T23:43:19.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.938+0000 7f06c95b8640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 0x7f06b00a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f06b4006fd0 tx=0x7f06b4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:19.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:19.942+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f06bc0bd470 con 0x7f06c4102250 2024-04-24T23:43:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.109+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f06bc0b93e0 con 0x7f06c4102250 2024-04-24T23:43:20.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.128+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06c4062340 con 0x7f06c4102250 2024-04-24T23:43:20.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.129+0000 7f06ba7fc640 1 -- 172.21.15.107:0/1988015121 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f06bc0e0ae0 con 0x7f06c4102250 2024-04-24T23:43:20.129 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:20.129 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:20.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 msgr2=0x7f06b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 0x7f06b00a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f06b4006fd0 tx=0x7f06b4008040 comp rx=0 tx=0).stop 2024-04-24T23:43:20.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 msgr2=0x7f06c419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06bc031ae0 tx=0x7f06bc031ee0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 shutdown_connections 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f06b00a2b90 0x7f06b00a5050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c4102250 0x7f06c419ee50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06c4101890 0x7f06c419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 --2- 172.21.15.107:0/1988015121 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f06c41006a0 0x7f06c406a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 >> 172.21.15.107:0/1988015121 conn(0x7f06c40fbc30 msgr2=0x7f06c4104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 shutdown_connections 2024-04-24T23:43:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.131+0000 7f06cb81c640 1 -- 172.21.15.107:0/1988015121 wait complete. 2024-04-24T23:43:20.143 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T23:43:20.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:43:20.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 -- 172.21.15.107:0/3668494877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101040 msgr2=0x7fd1141014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/3668494877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101040 0x7fd1141014c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd104009900 tx=0x7fd10402a410 comp rx=0 tx=0).stop 2024-04-24T23:43:20.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 -- 172.21.15.107:0/3668494877 shutdown_connections 2024-04-24T23:43:20.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/3668494877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101a00 0x7fd114108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/3668494877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101040 0x7fd1141014c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/3668494877 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 0x7fd114100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 -- 172.21.15.107:0/3668494877 >> 172.21.15.107:0/3668494877 conn(0x7fd1140fb420 msgr2=0x7fd1140fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 -- 172.21.15.107:0/3668494877 shutdown_connections 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.241+0000 7fd11c7c1640 1 -- 172.21.15.107:0/3668494877 wait complete. 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 Processor -- start 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 -- start start 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 0x7fd114197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101040 0x7fd1141980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410b330 con 0x7fd114101a00 2024-04-24T23:43:20.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410b1b0 con 0x7fd114101040 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11c7c1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410b4b0 con 0x7fd1140ffe60 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42590/0 (socket says 172.21.15.107:42590) 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 -- 172.21.15.107:0/894917940 learned_addr learned my addr 172.21.15.107:0/894917940 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11a55d640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 0x7fd114197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd119d5c640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101040 0x7fd1141980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 -- 172.21.15.107:0/894917940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 msgr2=0x7fd114197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 0x7fd114197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 -- 172.21.15.107:0/894917940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101040 msgr2=0x7fd1141980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101040 0x7fd1141980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 -- 172.21.15.107:0/894917940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11419f7d0 con 0x7fd114101a00 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd11ad5e640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00dd70 tx=0x7fd10c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.242+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c004d20 con 0x7fd114101a00 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.243+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd10c005500 con 0x7fd114101a00 2024-04-24T23:43:20.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.243+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11419fac0 con 0x7fd114101a00 2024-04-24T23:43:20.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.243+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c00ea80 con 0x7fd114101a00 2024-04-24T23:43:20.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.243+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd11419ff90 con 0x7fd114101a00 2024-04-24T23:43:20.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.243+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1141a0220 con 0x7fd114101a00 2024-04-24T23:43:20.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.244+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd10c0056b0 con 0x7fd114101a00 2024-04-24T23:43:20.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.245+0000 7fd10b7fe640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 0x7fd1000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.245+0000 7fd11a55d640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 0x7fd1000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.245+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd10c0cf850 con 0x7fd114101a00 2024-04-24T23:43:20.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.245+0000 7fd11a55d640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 0x7fd1000a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd114100bb0 tx=0x7fd110009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:20.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.250+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd10c09c520 con 0x7fd114101a00 2024-04-24T23:43:20.335 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: 2024-04-24T23:43:20.032+0000 7f0f86c79700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-24T23:43:20.335 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: 2024-04-24T23:43:20.032+0000 7f0f86c79700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:active 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:20 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1874735645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:43:20.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:active 2024-04-24T23:43:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-24T23:43:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:43:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:20 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1874735645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:43:20.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.423+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1141a03f0 con 0x7fd1000a2c20 2024-04-24T23:43:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.433+0000 7fd10b7fe640 1 -- 172.21.15.107:0/894917940 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340656 (secure 0 0 0) 0x7fd1141a03f0 con 0x7fd1000a2c20 2024-04-24T23:43:20.438 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 msgr2=0x7fd1000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 0x7fd1000a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd114100bb0 tx=0x7fd110009110 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 msgr2=0x7fd11419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00dd70 tx=0x7fd10c00b730 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 shutdown_connections 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1000a2c20 0x7fd1000a50e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd114101a00 0x7fd11419f160 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd114101040 0x7fd1141980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 --2- 172.21.15.107:0/894917940 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1140ffe60 0x7fd114197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 >> 172.21.15.107:0/894917940 conn(0x7fd1140fb420 msgr2=0x7fd114106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 shutdown_connections 2024-04-24T23:43:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.436+0000 7fd11c7c1640 1 -- 172.21.15.107:0/894917940 wait complete. 2024-04-24T23:43:20.447 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-24T23:43:19.979962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601077,"num_objects":91,"num_object_clones":0,"num_object_copies":275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":142,"num_read_kb":122,"num_write":368,"num_write_kb":3006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365084,"kb_used_data":11772,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576740,"statfs":{"total":1151940427776,"available":1151566581760,"internally_reserved":0,"allocated":12054528,"data_stored":7553197,"data_compressed":284682,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6210,"num_objects":67,"num_object_clones":0,"num_object_copies":203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":140,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406895+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:42:56.406895+0000","last_peered":"2024-04-24T23:42:56.406895+0000","last_clean":"2024-04-24T23:42:56.406895+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:42:56.406895+0000","last_undegraded":"2024-04-24T23:42:56.406895+0000","last_fullsized":"2024-04-24T23:42:56.406895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079728+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:42:57.079728+0000","last_peered":"2024-04-24T23:42:57.079728+0000","last_clean":"2024-04-24T23:42:57.079728+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:42:57.079728+0000","last_undegraded":"2024-04-24T23:42:57.079728+0000","last_fullsized":"2024-04-24T23:42:57.079728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079477+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:42:57.079477+0000","last_peered":"2024-04-24T23:42:57.079477+0000","last_clean":"2024-04-24T23:42:57.079477+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:42:57.079477+0000","last_undegraded":"2024-04-24T23:42:57.079477+0000","last_fullsized":"2024-04-24T23:42:57.079477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407113+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:42:56.407113+0000","last_peered":"2024-04-24T23:42:56.407113+0000","last_clean":"2024-04-24T23:42:56.407113+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:42:56.407113+0000","last_undegraded":"2024-04-24T23:42:56.407113+0000","last_fullsized":"2024-04-24T23:42:56.407113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364076+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:42:56.364076+0000","last_peered":"2024-04-24T23:42:56.364076+0000","last_clean":"2024-04-24T23:42:56.364076+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:42:56.364076+0000","last_undegraded":"2024-04-24T23:42:56.364076+0000","last_fullsized":"2024-04-24T23:42:56.364076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407154+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:42:56.407154+0000","last_peered":"2024-04-24T23:42:56.407154+0000","last_clean":"2024-04-24T23:42:56.407154+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:42:56.407154+0000","last_undegraded":"2024-04-24T23:42:56.407154+0000","last_fullsized":"2024-04-24T23:42:56.407154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313676+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:42:56.313676+0000","last_peered":"2024-04-24T23:42:56.313676+0000","last_clean":"2024-04-24T23:42:56.313676+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:42:56.313676+0000","last_undegraded":"2024-04-24T23:42:56.313676+0000","last_fullsized":"2024-04-24T23:42:56.313676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.405833+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:42:56.405833+0000","last_peered":"2024-04-24T23:42:56.405833+0000","last_clean":"2024-04-24T23:42:56.405833+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:42:56.405833+0000","last_undegraded":"2024-04-24T23:42:56.405833+0000","last_fullsized":"2024-04-24T23:42:56.405833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406820+0000","last_change":"2024-04-24T23:42:51.071718+0000","last_active":"2024-04-24T23:42:56.406820+0000","last_peered":"2024-04-24T23:42:56.406820+0000","last_clean":"2024-04-24T23:42:56.406820+0000","last_became_active":"2024-04-24T23:42:51.071371+0000","last_became_peered":"2024-04-24T23:42:51.071371+0000","last_unstale":"2024-04-24T23:42:56.406820+0000","last_undegraded":"2024-04-24T23:42:56.406820+0000","last_fullsized":"2024-04-24T23:42:56.406820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:04.897581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407948+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:42:56.407948+0000","last_peered":"2024-04-24T23:42:56.407948+0000","last_clean":"2024-04-24T23:42:56.407948+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:42:56.407948+0000","last_undegraded":"2024-04-24T23:42:56.407948+0000","last_fullsized":"2024-04-24T23:42:56.407948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314313+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:42:56.314313+0000","last_peered":"2024-04-24T23:42:56.314313+0000","last_clean":"2024-04-24T23:42:56.314313+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:42:56.314313+0000","last_undegraded":"2024-04-24T23:42:56.314313+0000","last_fullsized":"2024-04-24T23:42:56.314313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407069+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:42:56.407069+0000","last_peered":"2024-04-24T23:42:56.407069+0000","last_clean":"2024-04-24T23:42:56.407069+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:42:56.407069+0000","last_undegraded":"2024-04-24T23:42:56.407069+0000","last_fullsized":"2024-04-24T23:42:56.407069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406139+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:42:56.406139+0000","last_peered":"2024-04-24T23:42:56.406139+0000","last_clean":"2024-04-24T23:42:56.406139+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:42:56.406139+0000","last_undegraded":"2024-04-24T23:42:56.406139+0000","last_fullsized":"2024-04-24T23:42:56.406139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079331+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:42:57.079331+0000","last_peered":"2024-04-24T23:42:57.079331+0000","last_clean":"2024-04-24T23:42:57.079331+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:42:57.079331+0000","last_undegraded":"2024-04-24T23:42:57.079331+0000","last_fullsized":"2024-04-24T23:42:57.079331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079323+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:42:57.079323+0000","last_peered":"2024-04-24T23:42:57.079323+0000","last_clean":"2024-04-24T23:42:57.079323+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:42:57.079323+0000","last_undegraded":"2024-04-24T23:42:57.079323+0000","last_fullsized":"2024-04-24T23:42:57.079323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406225+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:42:56.406225+0000","last_peered":"2024-04-24T23:42:56.406225+0000","last_clean":"2024-04-24T23:42:56.406225+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:42:56.406225+0000","last_undegraded":"2024-04-24T23:42:56.406225+0000","last_fullsized":"2024-04-24T23:42:56.406225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406738+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:42:56.406738+0000","last_peered":"2024-04-24T23:42:56.406738+0000","last_clean":"2024-04-24T23:42:56.406738+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:42:56.406738+0000","last_undegraded":"2024-04-24T23:42:56.406738+0000","last_fullsized":"2024-04-24T23:42:56.406738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315627+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:42:56.315627+0000","last_peered":"2024-04-24T23:42:56.315627+0000","last_clean":"2024-04-24T23:42:56.315627+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:42:56.315627+0000","last_undegraded":"2024-04-24T23:42:56.315627+0000","last_fullsized":"2024-04-24T23:42:56.315627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314144+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:42:56.314144+0000","last_peered":"2024-04-24T23:42:56.314144+0000","last_clean":"2024-04-24T23:42:56.314144+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:42:56.314144+0000","last_undegraded":"2024-04-24T23:42:56.314144+0000","last_fullsized":"2024-04-24T23:42:56.314144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313864+0000","last_change":"2024-04-24T23:42:55.327166+0000","last_active":"2024-04-24T23:42:56.313864+0000","last_peered":"2024-04-24T23:42:56.313864+0000","last_clean":"2024-04-24T23:42:56.313864+0000","last_became_active":"2024-04-24T23:42:55.326708+0000","last_became_peered":"2024-04-24T23:42:55.326708+0000","last_unstale":"2024-04-24T23:42:56.313864+0000","last_undegraded":"2024-04-24T23:42:56.313864+0000","last_fullsized":"2024-04-24T23:42:56.313864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:39.604161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.084545+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:43:16.084545+0000","last_peered":"2024-04-24T23:43:16.084545+0000","last_clean":"2024-04-24T23:43:16.084545+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:43:16.084545+0000","last_undegraded":"2024-04-24T23:43:16.084545+0000","last_fullsized":"2024-04-24T23:43:16.084545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406594+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:42:56.406594+0000","last_peered":"2024-04-24T23:42:56.406594+0000","last_clean":"2024-04-24T23:42:56.406594+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:42:56.406594+0000","last_undegraded":"2024-04-24T23:42:56.406594+0000","last_fullsized":"2024-04-24T23:42:56.406594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407665+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:42:56.407665+0000","last_peered":"2024-04-24T23:42:56.407665+0000","last_clean":"2024-04-24T23:42:56.407665+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:42:56.407665+0000","last_undegraded":"2024-04-24T23:42:56.407665+0000","last_fullsized":"2024-04-24T23:42:56.407665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406834+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:42:56.406834+0000","last_peered":"2024-04-24T23:42:56.406834+0000","last_clean":"2024-04-24T23:42:56.406834+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:42:56.406834+0000","last_undegraded":"2024-04-24T23:42:56.406834+0000","last_fullsized":"2024-04-24T23:42:56.406834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079604+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:42:57.079604+0000","last_peered":"2024-04-24T23:42:57.079604+0000","last_clean":"2024-04-24T23:42:57.079604+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:42:57.079604+0000","last_undegraded":"2024-04-24T23:42:57.079604+0000","last_fullsized":"2024-04-24T23:42:57.079604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407229+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:42:56.407229+0000","last_peered":"2024-04-24T23:42:56.407229+0000","last_clean":"2024-04-24T23:42:56.407229+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:42:56.407229+0000","last_undegraded":"2024-04-24T23:42:56.407229+0000","last_fullsized":"2024-04-24T23:42:56.407229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.103137+0000","last_change":"2024-04-24T23:42:50.075128+0000","last_active":"2024-04-24T23:43:18.103137+0000","last_peered":"2024-04-24T23:43:18.103137+0000","last_clean":"2024-04-24T23:43:18.103137+0000","last_became_active":"2024-04-24T23:42:50.074583+0000","last_became_peered":"2024-04-24T23:42:50.074583+0000","last_unstale":"2024-04-24T23:43:18.103137+0000","last_undegraded":"2024-04-24T23:43:18.103137+0000","last_fullsized":"2024-04-24T23:43:18.103137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:52.857674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407550+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:42:56.407550+0000","last_peered":"2024-04-24T23:42:56.407550+0000","last_clean":"2024-04-24T23:42:56.407550+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:42:56.407550+0000","last_undegraded":"2024-04-24T23:42:56.407550+0000","last_fullsized":"2024-04-24T23:42:56.407550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406557+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:42:56.406557+0000","last_peered":"2024-04-24T23:42:56.406557+0000","last_clean":"2024-04-24T23:42:56.406557+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:42:56.406557+0000","last_undegraded":"2024-04-24T23:42:56.406557+0000","last_fullsized":"2024-04-24T23:42:56.406557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.100779+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:43:18.100779+0000","last_peered":"2024-04-24T23:43:18.100779+0000","last_clean":"2024-04-24T23:43:18.100779+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:43:18.100779+0000","last_undegraded":"2024-04-24T23:43:18.100779+0000","last_fullsized":"2024-04-24T23:43:18.100779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406977+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:42:56.406977+0000","last_peered":"2024-04-24T23:42:56.406977+0000","last_clean":"2024-04-24T23:42:56.406977+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:42:56.406977+0000","last_undegraded":"2024-04-24T23:42:56.406977+0000","last_fullsized":"2024-04-24T23:42:56.406977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314150+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:42:56.314150+0000","last_peered":"2024-04-24T23:42:56.314150+0000","last_clean":"2024-04-24T23:42:56.314150+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:42:56.314150+0000","last_undegraded":"2024-04-24T23:42:56.314150+0000","last_fullsized":"2024-04-24T23:42:56.314150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075846+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:43:15.075846+0000","last_peered":"2024-04-24T23:43:15.075846+0000","last_clean":"2024-04-24T23:43:15.075846+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:43:15.075846+0000","last_undegraded":"2024-04-24T23:43:15.075846+0000","last_fullsized":"2024-04-24T23:43:15.075846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.363982+0000","last_change":"2024-04-24T23:42:51.065658+0000","last_active":"2024-04-24T23:42:56.363982+0000","last_peered":"2024-04-24T23:42:56.363982+0000","last_clean":"2024-04-24T23:42:56.363982+0000","last_became_active":"2024-04-24T23:42:51.065295+0000","last_became_peered":"2024-04-24T23:42:51.065295+0000","last_unstale":"2024-04-24T23:42:56.363982+0000","last_undegraded":"2024-04-24T23:42:56.363982+0000","last_fullsized":"2024-04-24T23:42:56.363982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:17.536269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313996+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:42:56.313996+0000","last_peered":"2024-04-24T23:42:56.313996+0000","last_clean":"2024-04-24T23:42:56.313996+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:42:56.313996+0000","last_undegraded":"2024-04-24T23:42:56.313996+0000","last_fullsized":"2024-04-24T23:42:56.313996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407028+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:42:56.407028+0000","last_peered":"2024-04-24T23:42:56.407028+0000","last_clean":"2024-04-24T23:42:56.407028+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:42:56.407028+0000","last_undegraded":"2024-04-24T23:42:56.407028+0000","last_fullsized":"2024-04-24T23:42:56.407028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406962+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:42:56.406962+0000","last_peered":"2024-04-24T23:42:56.406962+0000","last_clean":"2024-04-24T23:42:56.406962+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:42:56.406962+0000","last_undegraded":"2024-04-24T23:42:56.406962+0000","last_fullsized":"2024-04-24T23:42:56.406962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406853+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:42:56.406853+0000","last_peered":"2024-04-24T23:42:56.406853+0000","last_clean":"2024-04-24T23:42:56.406853+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:42:56.406853+0000","last_undegraded":"2024-04-24T23:42:56.406853+0000","last_fullsized":"2024-04-24T23:42:56.406853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083897+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:43:16.083897+0000","last_peered":"2024-04-24T23:43:16.083897+0000","last_clean":"2024-04-24T23:43:16.083897+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:43:16.083897+0000","last_undegraded":"2024-04-24T23:43:16.083897+0000","last_fullsized":"2024-04-24T23:43:16.083897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406099+0000","last_change":"2024-04-24T23:42:51.073723+0000","last_active":"2024-04-24T23:42:56.406099+0000","last_peered":"2024-04-24T23:42:56.406099+0000","last_clean":"2024-04-24T23:42:56.406099+0000","last_became_active":"2024-04-24T23:42:51.073323+0000","last_became_peered":"2024-04-24T23:42:51.073323+0000","last_unstale":"2024-04-24T23:42:56.406099+0000","last_undegraded":"2024-04-24T23:42:56.406099+0000","last_fullsized":"2024-04-24T23:42:56.406099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:12.574032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079849+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:42:57.079849+0000","last_peered":"2024-04-24T23:42:57.079849+0000","last_clean":"2024-04-24T23:42:57.079849+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:42:57.079849+0000","last_undegraded":"2024-04-24T23:42:57.079849+0000","last_fullsized":"2024-04-24T23:42:57.079849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075584+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:43:15.075584+0000","last_peered":"2024-04-24T23:43:15.075584+0000","last_clean":"2024-04-24T23:43:15.075584+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:43:15.075584+0000","last_undegraded":"2024-04-24T23:43:15.075584+0000","last_fullsized":"2024-04-24T23:43:15.075584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315177+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:42:56.315177+0000","last_peered":"2024-04-24T23:42:56.315177+0000","last_clean":"2024-04-24T23:42:56.315177+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:42:56.315177+0000","last_undegraded":"2024-04-24T23:42:56.315177+0000","last_fullsized":"2024-04-24T23:42:56.315177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406559+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:42:56.406559+0000","last_peered":"2024-04-24T23:42:56.406559+0000","last_clean":"2024-04-24T23:42:56.406559+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:42:56.406559+0000","last_undegraded":"2024-04-24T23:42:56.406559+0000","last_fullsized":"2024-04-24T23:42:56.406559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.073446+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:43:15.073446+0000","last_peered":"2024-04-24T23:43:15.073446+0000","last_clean":"2024-04-24T23:43:15.073446+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:43:15.073446+0000","last_undegraded":"2024-04-24T23:43:15.073446+0000","last_fullsized":"2024-04-24T23:43:15.073446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406787+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:42:56.406787+0000","last_peered":"2024-04-24T23:42:56.406787+0000","last_clean":"2024-04-24T23:42:56.406787+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:42:56.406787+0000","last_undegraded":"2024-04-24T23:42:56.406787+0000","last_fullsized":"2024-04-24T23:42:56.406787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315124+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:42:56.315124+0000","last_peered":"2024-04-24T23:42:56.315124+0000","last_clean":"2024-04-24T23:42:56.315124+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:42:56.315124+0000","last_undegraded":"2024-04-24T23:42:56.315124+0000","last_fullsized":"2024-04-24T23:42:56.315124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.100837+0000","last_change":"2024-04-24T23:42:50.070204+0000","last_active":"2024-04-24T23:43:18.100837+0000","last_peered":"2024-04-24T23:43:18.100837+0000","last_clean":"2024-04-24T23:43:18.100837+0000","last_became_active":"2024-04-24T23:42:50.069708+0000","last_became_peered":"2024-04-24T23:42:50.069708+0000","last_unstale":"2024-04-24T23:43:18.100837+0000","last_undegraded":"2024-04-24T23:43:18.100837+0000","last_fullsized":"2024-04-24T23:43:18.100837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:55.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079542+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:42:57.079542+0000","last_peered":"2024-04-24T23:42:57.079542+0000","last_clean":"2024-04-24T23:42:57.079542+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:42:57.079542+0000","last_undegraded":"2024-04-24T23:42:57.079542+0000","last_fullsized":"2024-04-24T23:42:57.079542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407620+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:42:56.407620+0000","last_peered":"2024-04-24T23:42:56.407620+0000","last_clean":"2024-04-24T23:42:56.407620+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:42:56.407620+0000","last_undegraded":"2024-04-24T23:42:56.407620+0000","last_fullsized":"2024-04-24T23:42:56.407620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075307+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:43:15.075307+0000","last_peered":"2024-04-24T23:43:15.075307+0000","last_clean":"2024-04-24T23:43:15.075307+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:43:15.075307+0000","last_undegraded":"2024-04-24T23:43:15.075307+0000","last_fullsized":"2024-04-24T23:43:15.075307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407646+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:42:56.407646+0000","last_peered":"2024-04-24T23:42:56.407646+0000","last_clean":"2024-04-24T23:42:56.407646+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:42:56.407646+0000","last_undegraded":"2024-04-24T23:42:56.407646+0000","last_fullsized":"2024-04-24T23:42:56.407646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315179+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:42:56.315179+0000","last_peered":"2024-04-24T23:42:56.315179+0000","last_clean":"2024-04-24T23:42:56.315179+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:42:56.315179+0000","last_undegraded":"2024-04-24T23:42:56.315179+0000","last_fullsized":"2024-04-24T23:42:56.315179+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.074404+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:43:15.074404+0000","last_peered":"2024-04-24T23:43:15.074404+0000","last_clean":"2024-04-24T23:43:15.074404+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:43:15.074404+0000","last_undegraded":"2024-04-24T23:43:15.074404+0000","last_fullsized":"2024-04-24T23:43:15.074404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364279+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:42:56.364279+0000","last_peered":"2024-04-24T23:42:56.364279+0000","last_clean":"2024-04-24T23:42:56.364279+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:42:56.364279+0000","last_undegraded":"2024-04-24T23:42:56.364279+0000","last_fullsized":"2024-04-24T23:42:56.364279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407363+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:42:56.407363+0000","last_peered":"2024-04-24T23:42:56.407363+0000","last_clean":"2024-04-24T23:42:56.407363+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:42:56.407363+0000","last_undegraded":"2024-04-24T23:42:56.407363+0000","last_fullsized":"2024-04-24T23:42:56.407363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.100075+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:43:19.100075+0000","last_peered":"2024-04-24T23:43:19.100075+0000","last_clean":"2024-04-24T23:43:19.100075+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:43:19.100075+0000","last_undegraded":"2024-04-24T23:43:19.100075+0000","last_fullsized":"2024-04-24T23:43:19.100075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406365+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:42:56.406365+0000","last_peered":"2024-04-24T23:42:56.406365+0000","last_clean":"2024-04-24T23:42:56.406365+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:42:56.406365+0000","last_undegraded":"2024-04-24T23:42:56.406365+0000","last_fullsized":"2024-04-24T23:42:56.406365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406476+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:42:56.406476+0000","last_peered":"2024-04-24T23:42:56.406476+0000","last_clean":"2024-04-24T23:42:56.406476+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:42:56.406476+0000","last_undegraded":"2024-04-24T23:42:56.406476+0000","last_fullsized":"2024-04-24T23:42:56.406476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075778+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:43:15.075778+0000","last_peered":"2024-04-24T23:43:15.075778+0000","last_clean":"2024-04-24T23:43:15.075778+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:43:15.075778+0000","last_undegraded":"2024-04-24T23:43:15.075778+0000","last_fullsized":"2024-04-24T23:43:15.075778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314064+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:42:56.314064+0000","last_peered":"2024-04-24T23:42:56.314064+0000","last_clean":"2024-04-24T23:42:56.314064+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:42:56.314064+0000","last_undegraded":"2024-04-24T23:42:56.314064+0000","last_fullsized":"2024-04-24T23:42:56.314064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315673+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:42:56.315673+0000","last_peered":"2024-04-24T23:42:56.315673+0000","last_clean":"2024-04-24T23:42:56.315673+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:42:56.315673+0000","last_undegraded":"2024-04-24T23:42:56.315673+0000","last_fullsized":"2024-04-24T23:42:56.315673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.102764+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:43:19.102764+0000","last_peered":"2024-04-24T23:43:19.102764+0000","last_clean":"2024-04-24T23:43:19.102764+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:43:19.102764+0000","last_undegraded":"2024-04-24T23:43:19.102764+0000","last_fullsized":"2024-04-24T23:43:19.102764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314048+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:42:56.314048+0000","last_peered":"2024-04-24T23:42:56.314048+0000","last_clean":"2024-04-24T23:42:56.314048+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:42:56.314048+0000","last_undegraded":"2024-04-24T23:42:56.314048+0000","last_fullsized":"2024-04-24T23:42:56.314048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406838+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:42:56.406838+0000","last_peered":"2024-04-24T23:42:56.406838+0000","last_clean":"2024-04-24T23:42:56.406838+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:42:56.406838+0000","last_undegraded":"2024-04-24T23:42:56.406838+0000","last_fullsized":"2024-04-24T23:42:56.406838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.091068+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:43:17.091068+0000","last_peered":"2024-04-24T23:43:17.091068+0000","last_clean":"2024-04-24T23:43:17.091068+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:43:17.091068+0000","last_undegraded":"2024-04-24T23:43:17.091068+0000","last_fullsized":"2024-04-24T23:43:17.091068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407479+0000","last_change":"2024-04-24T23:42:51.076145+0000","last_active":"2024-04-24T23:42:56.407479+0000","last_peered":"2024-04-24T23:42:56.407479+0000","last_clean":"2024-04-24T23:42:56.407479+0000","last_became_active":"2024-04-24T23:42:51.073075+0000","last_became_peered":"2024-04-24T23:42:51.073075+0000","last_unstale":"2024-04-24T23:42:56.407479+0000","last_undegraded":"2024-04-24T23:42:56.407479+0000","last_fullsized":"2024-04-24T23:42:56.407479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:04.519183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407375+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:42:56.407375+0000","last_peered":"2024-04-24T23:42:56.407375+0000","last_clean":"2024-04-24T23:42:56.407375+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:42:56.407375+0000","last_undegraded":"2024-04-24T23:42:56.407375+0000","last_fullsized":"2024-04-24T23:42:56.407375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364318+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:42:56.364318+0000","last_peered":"2024-04-24T23:42:56.364318+0000","last_clean":"2024-04-24T23:42:56.364318+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:42:56.364318+0000","last_undegraded":"2024-04-24T23:42:56.364318+0000","last_fullsized":"2024-04-24T23:42:56.364318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406627+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:42:56.406627+0000","last_peered":"2024-04-24T23:42:56.406627+0000","last_clean":"2024-04-24T23:42:56.406627+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:42:56.406627+0000","last_undegraded":"2024-04-24T23:42:56.406627+0000","last_fullsized":"2024-04-24T23:42:56.406627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364128+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:42:56.364128+0000","last_peered":"2024-04-24T23:42:56.364128+0000","last_clean":"2024-04-24T23:42:56.364128+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:42:56.364128+0000","last_undegraded":"2024-04-24T23:42:56.364128+0000","last_fullsized":"2024-04-24T23:42:56.364128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315408+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:42:56.315408+0000","last_peered":"2024-04-24T23:42:56.315408+0000","last_clean":"2024-04-24T23:42:56.315408+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:42:56.315408+0000","last_undegraded":"2024-04-24T23:42:56.315408+0000","last_fullsized":"2024-04-24T23:42:56.315408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407196+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:42:56.407196+0000","last_peered":"2024-04-24T23:42:56.407196+0000","last_clean":"2024-04-24T23:42:56.407196+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:42:56.407196+0000","last_undegraded":"2024-04-24T23:42:56.407196+0000","last_fullsized":"2024-04-24T23:42:56.407196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407024+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:42:56.407024+0000","last_peered":"2024-04-24T23:42:56.407024+0000","last_clean":"2024-04-24T23:42:56.407024+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:42:56.407024+0000","last_undegraded":"2024-04-24T23:42:56.407024+0000","last_fullsized":"2024-04-24T23:42:56.407024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.101679+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:43:19.101679+0000","last_peered":"2024-04-24T23:43:19.101679+0000","last_clean":"2024-04-24T23:43:19.101679+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:43:19.101679+0000","last_undegraded":"2024-04-24T23:43:19.101679+0000","last_fullsized":"2024-04-24T23:43:19.101679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407472+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:42:56.407472+0000","last_peered":"2024-04-24T23:42:56.407472+0000","last_clean":"2024-04-24T23:42:56.407472+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:42:56.407472+0000","last_undegraded":"2024-04-24T23:42:56.407472+0000","last_fullsized":"2024-04-24T23:42:56.407472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406473+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:42:56.406473+0000","last_peered":"2024-04-24T23:42:56.406473+0000","last_clean":"2024-04-24T23:42:56.406473+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:42:56.406473+0000","last_undegraded":"2024-04-24T23:42:56.406473+0000","last_fullsized":"2024-04-24T23:42:56.406473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406521+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:42:56.406521+0000","last_peered":"2024-04-24T23:42:56.406521+0000","last_clean":"2024-04-24T23:42:56.406521+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:42:56.406521+0000","last_undegraded":"2024-04-24T23:42:56.406521+0000","last_fullsized":"2024-04-24T23:42:56.406521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083222+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:43:16.083222+0000","last_peered":"2024-04-24T23:43:16.083222+0000","last_clean":"2024-04-24T23:43:16.083222+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:43:16.083222+0000","last_undegraded":"2024-04-24T23:43:16.083222+0000","last_fullsized":"2024-04-24T23:43:16.083222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314092+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:42:56.314092+0000","last_peered":"2024-04-24T23:42:56.314092+0000","last_clean":"2024-04-24T23:42:56.314092+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:42:56.314092+0000","last_undegraded":"2024-04-24T23:42:56.314092+0000","last_fullsized":"2024-04-24T23:42:56.314092+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406005+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:42:56.406005+0000","last_peered":"2024-04-24T23:42:56.406005+0000","last_clean":"2024-04-24T23:42:56.406005+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:42:56.406005+0000","last_undegraded":"2024-04-24T23:42:56.406005+0000","last_fullsized":"2024-04-24T23:42:56.406005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.099769+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:43:19.099769+0000","last_peered":"2024-04-24T23:43:19.099769+0000","last_clean":"2024-04-24T23:43:19.099769+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:43:19.099769+0000","last_undegraded":"2024-04-24T23:43:19.099769+0000","last_fullsized":"2024-04-24T23:43:19.099769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313570+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:42:56.313570+0000","last_peered":"2024-04-24T23:42:56.313570+0000","last_clean":"2024-04-24T23:42:56.313570+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:42:56.313570+0000","last_undegraded":"2024-04-24T23:42:56.313570+0000","last_fullsized":"2024-04-24T23:42:56.313570+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407419+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:42:56.407419+0000","last_peered":"2024-04-24T23:42:56.407419+0000","last_clean":"2024-04-24T23:42:56.407419+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:42:56.407419+0000","last_undegraded":"2024-04-24T23:42:56.407419+0000","last_fullsized":"2024-04-24T23:42:56.407419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.405959+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:42:56.405959+0000","last_peered":"2024-04-24T23:42:56.405959+0000","last_clean":"2024-04-24T23:42:56.405959+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:42:56.405959+0000","last_undegraded":"2024-04-24T23:42:56.405959+0000","last_fullsized":"2024-04-24T23:42:56.405959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083796+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:43:16.083796+0000","last_peered":"2024-04-24T23:43:16.083796+0000","last_clean":"2024-04-24T23:43:16.083796+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:43:16.083796+0000","last_undegraded":"2024-04-24T23:43:16.083796+0000","last_fullsized":"2024-04-24T23:43:16.083796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406453+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:42:56.406453+0000","last_peered":"2024-04-24T23:42:56.406453+0000","last_clean":"2024-04-24T23:42:56.406453+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:42:56.406453+0000","last_undegraded":"2024-04-24T23:42:56.406453+0000","last_fullsized":"2024-04-24T23:42:56.406453+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.091142+0000","last_change":"2024-04-24T23:42:50.074409+0000","last_active":"2024-04-24T23:43:17.091142+0000","last_peered":"2024-04-24T23:43:17.091142+0000","last_clean":"2024-04-24T23:43:17.091142+0000","last_became_active":"2024-04-24T23:42:50.072784+0000","last_became_peered":"2024-04-24T23:42:50.072784+0000","last_unstale":"2024-04-24T23:43:17.091142+0000","last_undegraded":"2024-04-24T23:43:17.091142+0000","last_fullsized":"2024-04-24T23:43:17.091142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:08.625302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407062+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:42:56.407062+0000","last_peered":"2024-04-24T23:42:56.407062+0000","last_clean":"2024-04-24T23:42:56.407062+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:42:56.407062+0000","last_undegraded":"2024-04-24T23:42:56.407062+0000","last_fullsized":"2024-04-24T23:42:56.407062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406056+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:42:56.406056+0000","last_peered":"2024-04-24T23:42:56.406056+0000","last_clean":"2024-04-24T23:42:56.406056+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:42:56.406056+0000","last_undegraded":"2024-04-24T23:42:56.406056+0000","last_fullsized":"2024-04-24T23:42:56.406056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083767+0000","last_change":"2024-04-24T23:42:50.068569+0000","last_active":"2024-04-24T23:43:16.083767+0000","last_peered":"2024-04-24T23:43:16.083767+0000","last_clean":"2024-04-24T23:43:16.083767+0000","last_became_active":"2024-04-24T23:42:50.068078+0000","last_became_peered":"2024-04-24T23:42:50.068078+0000","last_unstale":"2024-04-24T23:43:16.083767+0000","last_undegraded":"2024-04-24T23:43:16.083767+0000","last_fullsized":"2024-04-24T23:43:16.083767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:48:15.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406635+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:42:56.406635+0000","last_peered":"2024-04-24T23:42:56.406635+0000","last_clean":"2024-04-24T23:42:56.406635+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:42:56.406635+0000","last_undegraded":"2024-04-24T23:42:56.406635+0000","last_fullsized":"2024-04-24T23:42:56.406635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313761+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:42:56.313761+0000","last_peered":"2024-04-24T23:42:56.313761+0000","last_clean":"2024-04-24T23:42:56.313761+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:42:56.313761+0000","last_undegraded":"2024-04-24T23:42:56.313761+0000","last_fullsized":"2024-04-24T23:42:56.313761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.102825+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:43:18.102825+0000","last_peered":"2024-04-24T23:43:18.102825+0000","last_clean":"2024-04-24T23:43:18.102825+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:43:18.102825+0000","last_undegraded":"2024-04-24T23:43:18.102825+0000","last_fullsized":"2024-04-24T23:43:18.102825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079788+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:42:57.079788+0000","last_peered":"2024-04-24T23:42:57.079788+0000","last_clean":"2024-04-24T23:42:57.079788+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:42:57.079788+0000","last_undegraded":"2024-04-24T23:42:57.079788+0000","last_fullsized":"2024-04-24T23:42:57.079788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407467+0000","last_change":"2024-04-24T23:42:55.322284+0000","last_active":"2024-04-24T23:42:56.407467+0000","last_peered":"2024-04-24T23:42:56.407467+0000","last_clean":"2024-04-24T23:42:56.407467+0000","last_became_active":"2024-04-24T23:42:55.321228+0000","last_became_peered":"2024-04-24T23:42:55.321228+0000","last_unstale":"2024-04-24T23:42:56.407467+0000","last_undegraded":"2024-04-24T23:42:56.407467+0000","last_fullsized":"2024-04-24T23:42:56.407467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:20.864377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.074256+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:43:15.074256+0000","last_peered":"2024-04-24T23:43:15.074256+0000","last_clean":"2024-04-24T23:43:15.074256+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:43:15.074256+0000","last_undegraded":"2024-04-24T23:43:15.074256+0000","last_fullsized":"2024-04-24T23:43:15.074256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.363820+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:42:56.363820+0000","last_peered":"2024-04-24T23:42:56.363820+0000","last_clean":"2024-04-24T23:42:56.363820+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:42:56.363820+0000","last_undegraded":"2024-04-24T23:42:56.363820+0000","last_fullsized":"2024-04-24T23:42:56.363820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315779+0000","last_change":"2024-04-24T23:42:55.324683+0000","last_active":"2024-04-24T23:42:56.315779+0000","last_peered":"2024-04-24T23:42:56.315779+0000","last_clean":"2024-04-24T23:42:56.315779+0000","last_became_active":"2024-04-24T23:42:55.324311+0000","last_became_peered":"2024-04-24T23:42:55.324311+0000","last_unstale":"2024-04-24T23:42:56.315779+0000","last_undegraded":"2024-04-24T23:42:56.315779+0000","last_fullsized":"2024-04-24T23:42:56.315779+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:17.659811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.102158+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:43:18.102158+0000","last_peered":"2024-04-24T23:43:18.102158+0000","last_clean":"2024-04-24T23:43:18.102158+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:43:18.102158+0000","last_undegraded":"2024-04-24T23:43:18.102158+0000","last_fullsized":"2024-04-24T23:43:18.102158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406114+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:42:56.406114+0000","last_peered":"2024-04-24T23:42:56.406114+0000","last_clean":"2024-04-24T23:42:56.406114+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:42:56.406114+0000","last_undegraded":"2024-04-24T23:42:56.406114+0000","last_fullsized":"2024-04-24T23:42:56.406114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:59.181180+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:42:59.181180+0000","last_peered":"2024-04-24T23:42:59.181180+0000","last_clean":"2024-04-24T23:42:59.181180+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:42:59.181180+0000","last_undegraded":"2024-04-24T23:42:59.181180+0000","last_fullsized":"2024-04-24T23:42:59.181180+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.090859+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:43:17.090859+0000","last_peered":"2024-04-24T23:43:17.090859+0000","last_clean":"2024-04-24T23:43:17.090859+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:43:17.090859+0000","last_undegraded":"2024-04-24T23:43:17.090859+0000","last_fullsized":"2024-04-24T23:43:17.090859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406511+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:42:56.406511+0000","last_peered":"2024-04-24T23:42:56.406511+0000","last_clean":"2024-04-24T23:42:56.406511+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:42:56.406511+0000","last_undegraded":"2024-04-24T23:42:56.406511+0000","last_fullsized":"2024-04-24T23:42:56.406511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407495+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:42:56.407495+0000","last_peered":"2024-04-24T23:42:56.407495+0000","last_clean":"2024-04-24T23:42:56.407495+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:42:56.407495+0000","last_undegraded":"2024-04-24T23:42:56.407495+0000","last_fullsized":"2024-04-24T23:42:56.407495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.073685+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:43:15.073685+0000","last_peered":"2024-04-24T23:43:15.073685+0000","last_clean":"2024-04-24T23:43:15.073685+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:43:15.073685+0000","last_undegraded":"2024-04-24T23:43:15.073685+0000","last_fullsized":"2024-04-24T23:43:15.073685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313952+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:42:56.313952+0000","last_peered":"2024-04-24T23:42:56.313952+0000","last_clean":"2024-04-24T23:42:56.313952+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:42:56.313952+0000","last_undegraded":"2024-04-24T23:42:56.313952+0000","last_fullsized":"2024-04-24T23:42:56.313952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407041+0000","last_change":"2024-04-24T23:42:55.319168+0000","last_active":"2024-04-24T23:42:56.407041+0000","last_peered":"2024-04-24T23:42:56.407041+0000","last_clean":"2024-04-24T23:42:56.407041+0000","last_became_active":"2024-04-24T23:42:55.318752+0000","last_became_peered":"2024-04-24T23:42:55.318752+0000","last_unstale":"2024-04-24T23:42:56.407041+0000","last_undegraded":"2024-04-24T23:42:56.407041+0000","last_fullsized":"2024-04-24T23:42:56.407041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:14.923700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.102925+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:43:19.102925+0000","last_peered":"2024-04-24T23:43:19.102925+0000","last_clean":"2024-04-24T23:43:19.102925+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:43:19.102925+0000","last_undegraded":"2024-04-24T23:43:19.102925+0000","last_fullsized":"2024-04-24T23:43:19.102925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314322+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:42:56.314322+0000","last_peered":"2024-04-24T23:42:56.314322+0000","last_clean":"2024-04-24T23:42:56.314322+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:42:56.314322+0000","last_undegraded":"2024-04-24T23:42:56.314322+0000","last_fullsized":"2024-04-24T23:42:56.314322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079486+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:42:57.079486+0000","last_peered":"2024-04-24T23:42:57.079486+0000","last_clean":"2024-04-24T23:42:57.079486+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:42:57.079486+0000","last_undegraded":"2024-04-24T23:42:57.079486+0000","last_fullsized":"2024-04-24T23:42:57.079486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407086+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:42:56.407086+0000","last_peered":"2024-04-24T23:42:56.407086+0000","last_clean":"2024-04-24T23:42:56.407086+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:42:56.407086+0000","last_undegraded":"2024-04-24T23:42:56.407086+0000","last_fullsized":"2024-04-24T23:42:56.407086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314247+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:42:56.314247+0000","last_peered":"2024-04-24T23:42:56.314247+0000","last_clean":"2024-04-24T23:42:56.314247+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:42:56.314247+0000","last_undegraded":"2024-04-24T23:42:56.314247+0000","last_fullsized":"2024-04-24T23:42:56.314247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406216+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:42:56.406216+0000","last_peered":"2024-04-24T23:42:56.406216+0000","last_clean":"2024-04-24T23:42:56.406216+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:42:56.406216+0000","last_undegraded":"2024-04-24T23:42:56.406216+0000","last_fullsized":"2024-04-24T23:42:56.406216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.099083+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:43:18.099083+0000","last_peered":"2024-04-24T23:43:18.099083+0000","last_clean":"2024-04-24T23:43:18.099083+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:43:18.099083+0000","last_undegraded":"2024-04-24T23:43:18.099083+0000","last_fullsized":"2024-04-24T23:43:18.099083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314311+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:42:56.314311+0000","last_peered":"2024-04-24T23:42:56.314311+0000","last_clean":"2024-04-24T23:42:56.314311+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:42:56.314311+0000","last_undegraded":"2024-04-24T23:42:56.314311+0000","last_fullsized":"2024-04-24T23:42:56.314311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406181+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:42:56.406181+0000","last_peered":"2024-04-24T23:42:56.406181+0000","last_clean":"2024-04-24T23:42:56.406181+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:42:56.406181+0000","last_undegraded":"2024-04-24T23:42:56.406181+0000","last_fullsized":"2024-04-24T23:42:56.406181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313579+0000","last_change":"2024-04-24T23:42:51.081821+0000","last_active":"2024-04-24T23:42:56.313579+0000","last_peered":"2024-04-24T23:42:56.313579+0000","last_clean":"2024-04-24T23:42:56.313579+0000","last_became_active":"2024-04-24T23:42:51.081246+0000","last_became_peered":"2024-04-24T23:42:51.081246+0000","last_unstale":"2024-04-24T23:42:56.313579+0000","last_undegraded":"2024-04-24T23:42:56.313579+0000","last_fullsized":"2024-04-24T23:42:56.313579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:24.375428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313501+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:42:56.313501+0000","last_peered":"2024-04-24T23:42:56.313501+0000","last_clean":"2024-04-24T23:42:56.313501+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:42:56.313501+0000","last_undegraded":"2024-04-24T23:42:56.313501+0000","last_fullsized":"2024-04-24T23:42:56.313501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079503+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:42:57.079503+0000","last_peered":"2024-04-24T23:42:57.079503+0000","last_clean":"2024-04-24T23:42:57.079503+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:42:57.079503+0000","last_undegraded":"2024-04-24T23:42:57.079503+0000","last_fullsized":"2024-04-24T23:42:57.079503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406976+0000","last_change":"2024-04-24T23:42:51.070156+0000","last_active":"2024-04-24T23:42:56.406976+0000","last_peered":"2024-04-24T23:42:56.406976+0000","last_clean":"2024-04-24T23:42:56.406976+0000","last_became_active":"2024-04-24T23:42:51.069842+0000","last_became_peered":"2024-04-24T23:42:51.069842+0000","last_unstale":"2024-04-24T23:42:56.406976+0000","last_undegraded":"2024-04-24T23:42:56.406976+0000","last_fullsized":"2024-04-24T23:42:56.406976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:06.133128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406728+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:42:56.406728+0000","last_peered":"2024-04-24T23:42:56.406728+0000","last_clean":"2024-04-24T23:42:56.406728+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:42:56.406728+0000","last_undegraded":"2024-04-24T23:42:56.406728+0000","last_fullsized":"2024-04-24T23:42:56.406728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314234+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:42:56.314234+0000","last_peered":"2024-04-24T23:42:56.314234+0000","last_clean":"2024-04-24T23:42:56.314234+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:42:56.314234+0000","last_undegraded":"2024-04-24T23:42:56.314234+0000","last_fullsized":"2024-04-24T23:42:56.314234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406728+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:42:56.406728+0000","last_peered":"2024-04-24T23:42:56.406728+0000","last_clean":"2024-04-24T23:42:56.406728+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:42:56.406728+0000","last_undegraded":"2024-04-24T23:42:56.406728+0000","last_fullsized":"2024-04-24T23:42:56.406728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406407+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:42:56.406407+0000","last_peered":"2024-04-24T23:42:56.406407+0000","last_clean":"2024-04-24T23:42:56.406407+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:42:56.406407+0000","last_undegraded":"2024-04-24T23:42:56.406407+0000","last_fullsized":"2024-04-24T23:42:56.406407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406711+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:42:56.406711+0000","last_peered":"2024-04-24T23:42:56.406711+0000","last_clean":"2024-04-24T23:42:56.406711+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:42:56.406711+0000","last_undegraded":"2024-04-24T23:42:56.406711+0000","last_fullsized":"2024-04-24T23:42:56.406711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407357+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:42:56.407357+0000","last_peered":"2024-04-24T23:42:56.407357+0000","last_clean":"2024-04-24T23:42:56.407357+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:42:56.407357+0000","last_undegraded":"2024-04-24T23:42:56.407357+0000","last_fullsized":"2024-04-24T23:42:56.407357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406237+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:42:56.406237+0000","last_peered":"2024-04-24T23:42:56.406237+0000","last_clean":"2024-04-24T23:42:56.406237+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:42:56.406237+0000","last_undegraded":"2024-04-24T23:42:56.406237+0000","last_fullsized":"2024-04-24T23:42:56.406237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406432+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:42:56.406432+0000","last_peered":"2024-04-24T23:42:56.406432+0000","last_clean":"2024-04-24T23:42:56.406432+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:42:56.406432+0000","last_undegraded":"2024-04-24T23:42:56.406432+0000","last_fullsized":"2024-04-24T23:42:56.406432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10475,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":185,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":64673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":478407,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478715,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":1079308,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477937,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479281,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":502328,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477623,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478306,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1069821,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:43:20.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:43:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] up:active 2024-04-24T23:43:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-24T23:43:20.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:43:20.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:20.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:20 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1874735645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 -- 172.21.15.107:0/1658874936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac102980 msgr2=0x7f0fac102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 --2- 172.21.15.107:0/1658874936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac102980 0x7f0fac102e00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009900 tx=0x7f0f9c02a410 comp rx=0 tx=0).stop 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 -- 172.21.15.107:0/1658874936 shutdown_connections 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 --2- 172.21.15.107:0/1658874936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fac103340 0x7f0fac109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 --2- 172.21.15.107:0/1658874936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac102980 0x7f0fac102e00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 --2- 172.21.15.107:0/1658874936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 0x7f0fac101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.587+0000 7f0fb425d640 1 -- 172.21.15.107:0/1658874936 >> 172.21.15.107:0/1658874936 conn(0x7f0fac0fcd20 msgr2=0x7f0fac0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- 172.21.15.107:0/1658874936 shutdown_connections 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- 172.21.15.107:0/1658874936 wait complete. 2024-04-24T23:43:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 Processor -- start 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- start start 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 0x7f0fac0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fac102980 0x7f0fac073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac10d410 con 0x7f0fac103340 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac10d290 con 0x7f0fac101790 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb425d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac10d590 con 0x7f0fac102980 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb27fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb27fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42600/0 (socket says 172.21.15.107:42600) 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb27fa640 1 -- 172.21.15.107:0/495487098 learned_addr learned my addr 172.21.15.107:0/495487098 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.588+0000 7f0fb1ff9640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 0x7f0fac0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 -- 172.21.15.107:0/495487098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fac102980 msgr2=0x7f0fac073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fac102980 0x7f0fac073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 -- 172.21.15.107:0/495487098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 msgr2=0x7f0fac0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 0x7f0fac0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 -- 172.21.15.107:0/495487098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac074900 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb27fa640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400cad0 tx=0x7f0fa400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400ae20 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fac1a1be0 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fa4005020 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400f6b0 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.589+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fac1a2180 con 0x7f0fac103340 2024-04-24T23:43:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.590+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fac108b30 con 0x7f0fac103340 2024-04-24T23:43:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.591+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0fa40051d0 con 0x7f0fac103340 2024-04-24T23:43:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.591+0000 7f0fa2ffd640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 0x7f0f980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.592+0000 7f0fb1ff9640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 0x7f0f980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.592+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0fa40cf7e0 con 0x7f0fac103340 2024-04-24T23:43:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.592+0000 7f0fb1ff9640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 0x7f0f980a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fac1027e0 tx=0x7f0fa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:20.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.596+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0fa4014030 con 0x7f0fac103340 2024-04-24T23:43:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.769+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0fac061b80 con 0x7f0f980a2c20 2024-04-24T23:43:20.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.778+0000 7f0fa2ffd640 1 -- 172.21.15.107:0/495487098 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340656 (secure 0 0 0) 0x7f0fac061b80 con 0x7f0f980a2c20 2024-04-24T23:43:20.782 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:20.785 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:43:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 msgr2=0x7f0f980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 0x7f0f980a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fac1027e0 tx=0x7f0fa8005f90 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 msgr2=0x7f0fac074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400cad0 tx=0x7f0fa400cf90 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 shutdown_connections 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f980a2c20 0x7f0f980a50e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fac103340 0x7f0fac074070 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0fac102980 0x7f0fac073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 --2- 172.21.15.107:0/495487098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fac101790 0x7f0fac0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.781+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 >> 172.21.15.107:0/495487098 conn(0x7f0fac0fcd20 msgr2=0x7f0fac105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.782+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 shutdown_connections 2024-04-24T23:43:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:20.782+0000 7f0fb425d640 1 -- 172.21.15.107:0/495487098 wait complete. 2024-04-24T23:43:20.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-24T23:43:19.979962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601077,"num_objects":91,"num_object_clones":0,"num_object_copies":275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":142,"num_read_kb":122,"num_write":368,"num_write_kb":3006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365084,"kb_used_data":11772,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576740,"statfs":{"total":1151940427776,"available":1151566581760,"internally_reserved":0,"allocated":12054528,"data_stored":7553197,"data_compressed":284682,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6210,"num_objects":67,"num_object_clones":0,"num_object_copies":203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":140,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406895+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:42:56.406895+0000","last_peered":"2024-04-24T23:42:56.406895+0000","last_clean":"2024-04-24T23:42:56.406895+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:42:56.406895+0000","last_undegraded":"2024-04-24T23:42:56.406895+0000","last_fullsized":"2024-04-24T23:42:56.406895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079728+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:42:57.079728+0000","last_peered":"2024-04-24T23:42:57.079728+0000","last_clean":"2024-04-24T23:42:57.079728+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:42:57.079728+0000","last_undegraded":"2024-04-24T23:42:57.079728+0000","last_fullsized":"2024-04-24T23:42:57.079728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079477+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:42:57.079477+0000","last_peered":"2024-04-24T23:42:57.079477+0000","last_clean":"2024-04-24T23:42:57.079477+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:42:57.079477+0000","last_undegraded":"2024-04-24T23:42:57.079477+0000","last_fullsized":"2024-04-24T23:42:57.079477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407113+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:42:56.407113+0000","last_peered":"2024-04-24T23:42:56.407113+0000","last_clean":"2024-04-24T23:42:56.407113+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:42:56.407113+0000","last_undegraded":"2024-04-24T23:42:56.407113+0000","last_fullsized":"2024-04-24T23:42:56.407113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364076+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:42:56.364076+0000","last_peered":"2024-04-24T23:42:56.364076+0000","last_clean":"2024-04-24T23:42:56.364076+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:42:56.364076+0000","last_undegraded":"2024-04-24T23:42:56.364076+0000","last_fullsized":"2024-04-24T23:42:56.364076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407154+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:42:56.407154+0000","last_peered":"2024-04-24T23:42:56.407154+0000","last_clean":"2024-04-24T23:42:56.407154+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:42:56.407154+0000","last_undegraded":"2024-04-24T23:42:56.407154+0000","last_fullsized":"2024-04-24T23:42:56.407154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313676+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:42:56.313676+0000","last_peered":"2024-04-24T23:42:56.313676+0000","last_clean":"2024-04-24T23:42:56.313676+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:42:56.313676+0000","last_undegraded":"2024-04-24T23:42:56.313676+0000","last_fullsized":"2024-04-24T23:42:56.313676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.405833+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:42:56.405833+0000","last_peered":"2024-04-24T23:42:56.405833+0000","last_clean":"2024-04-24T23:42:56.405833+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:42:56.405833+0000","last_undegraded":"2024-04-24T23:42:56.405833+0000","last_fullsized":"2024-04-24T23:42:56.405833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406820+0000","last_change":"2024-04-24T23:42:51.071718+0000","last_active":"2024-04-24T23:42:56.406820+0000","last_peered":"2024-04-24T23:42:56.406820+0000","last_clean":"2024-04-24T23:42:56.406820+0000","last_became_active":"2024-04-24T23:42:51.071371+0000","last_became_peered":"2024-04-24T23:42:51.071371+0000","last_unstale":"2024-04-24T23:42:56.406820+0000","last_undegraded":"2024-04-24T23:42:56.406820+0000","last_fullsized":"2024-04-24T23:42:56.406820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:04.897581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407948+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:42:56.407948+0000","last_peered":"2024-04-24T23:42:56.407948+0000","last_clean":"2024-04-24T23:42:56.407948+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:42:56.407948+0000","last_undegraded":"2024-04-24T23:42:56.407948+0000","last_fullsized":"2024-04-24T23:42:56.407948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314313+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:42:56.314313+0000","last_peered":"2024-04-24T23:42:56.314313+0000","last_clean":"2024-04-24T23:42:56.314313+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:42:56.314313+0000","last_undegraded":"2024-04-24T23:42:56.314313+0000","last_fullsized":"2024-04-24T23:42:56.314313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407069+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:42:56.407069+0000","last_peered":"2024-04-24T23:42:56.407069+0000","last_clean":"2024-04-24T23:42:56.407069+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:42:56.407069+0000","last_undegraded":"2024-04-24T23:42:56.407069+0000","last_fullsized":"2024-04-24T23:42:56.407069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406139+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:42:56.406139+0000","last_peered":"2024-04-24T23:42:56.406139+0000","last_clean":"2024-04-24T23:42:56.406139+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:42:56.406139+0000","last_undegraded":"2024-04-24T23:42:56.406139+0000","last_fullsized":"2024-04-24T23:42:56.406139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079331+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:42:57.079331+0000","last_peered":"2024-04-24T23:42:57.079331+0000","last_clean":"2024-04-24T23:42:57.079331+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:42:57.079331+0000","last_undegraded":"2024-04-24T23:42:57.079331+0000","last_fullsized":"2024-04-24T23:42:57.079331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079323+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:42:57.079323+0000","last_peered":"2024-04-24T23:42:57.079323+0000","last_clean":"2024-04-24T23:42:57.079323+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:42:57.079323+0000","last_undegraded":"2024-04-24T23:42:57.079323+0000","last_fullsized":"2024-04-24T23:42:57.079323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406225+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:42:56.406225+0000","last_peered":"2024-04-24T23:42:56.406225+0000","last_clean":"2024-04-24T23:42:56.406225+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:42:56.406225+0000","last_undegraded":"2024-04-24T23:42:56.406225+0000","last_fullsized":"2024-04-24T23:42:56.406225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406738+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:42:56.406738+0000","last_peered":"2024-04-24T23:42:56.406738+0000","last_clean":"2024-04-24T23:42:56.406738+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:42:56.406738+0000","last_undegraded":"2024-04-24T23:42:56.406738+0000","last_fullsized":"2024-04-24T23:42:56.406738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315627+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:42:56.315627+0000","last_peered":"2024-04-24T23:42:56.315627+0000","last_clean":"2024-04-24T23:42:56.315627+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:42:56.315627+0000","last_undegraded":"2024-04-24T23:42:56.315627+0000","last_fullsized":"2024-04-24T23:42:56.315627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314144+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:42:56.314144+0000","last_peered":"2024-04-24T23:42:56.314144+0000","last_clean":"2024-04-24T23:42:56.314144+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:42:56.314144+0000","last_undegraded":"2024-04-24T23:42:56.314144+0000","last_fullsized":"2024-04-24T23:42:56.314144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313864+0000","last_change":"2024-04-24T23:42:55.327166+0000","last_active":"2024-04-24T23:42:56.313864+0000","last_peered":"2024-04-24T23:42:56.313864+0000","last_clean":"2024-04-24T23:42:56.313864+0000","last_became_active":"2024-04-24T23:42:55.326708+0000","last_became_peered":"2024-04-24T23:42:55.326708+0000","last_unstale":"2024-04-24T23:42:56.313864+0000","last_undegraded":"2024-04-24T23:42:56.313864+0000","last_fullsized":"2024-04-24T23:42:56.313864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:39.604161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.084545+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:43:16.084545+0000","last_peered":"2024-04-24T23:43:16.084545+0000","last_clean":"2024-04-24T23:43:16.084545+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:43:16.084545+0000","last_undegraded":"2024-04-24T23:43:16.084545+0000","last_fullsized":"2024-04-24T23:43:16.084545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406594+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:42:56.406594+0000","last_peered":"2024-04-24T23:42:56.406594+0000","last_clean":"2024-04-24T23:42:56.406594+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:42:56.406594+0000","last_undegraded":"2024-04-24T23:42:56.406594+0000","last_fullsized":"2024-04-24T23:42:56.406594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407665+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:42:56.407665+0000","last_peered":"2024-04-24T23:42:56.407665+0000","last_clean":"2024-04-24T23:42:56.407665+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:42:56.407665+0000","last_undegraded":"2024-04-24T23:42:56.407665+0000","last_fullsized":"2024-04-24T23:42:56.407665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406834+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:42:56.406834+0000","last_peered":"2024-04-24T23:42:56.406834+0000","last_clean":"2024-04-24T23:42:56.406834+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:42:56.406834+0000","last_undegraded":"2024-04-24T23:42:56.406834+0000","last_fullsized":"2024-04-24T23:42:56.406834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079604+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:42:57.079604+0000","last_peered":"2024-04-24T23:42:57.079604+0000","last_clean":"2024-04-24T23:42:57.079604+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:42:57.079604+0000","last_undegraded":"2024-04-24T23:42:57.079604+0000","last_fullsized":"2024-04-24T23:42:57.079604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407229+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:42:56.407229+0000","last_peered":"2024-04-24T23:42:56.407229+0000","last_clean":"2024-04-24T23:42:56.407229+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:42:56.407229+0000","last_undegraded":"2024-04-24T23:42:56.407229+0000","last_fullsized":"2024-04-24T23:42:56.407229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.103137+0000","last_change":"2024-04-24T23:42:50.075128+0000","last_active":"2024-04-24T23:43:18.103137+0000","last_peered":"2024-04-24T23:43:18.103137+0000","last_clean":"2024-04-24T23:43:18.103137+0000","last_became_active":"2024-04-24T23:42:50.074583+0000","last_became_peered":"2024-04-24T23:42:50.074583+0000","last_unstale":"2024-04-24T23:43:18.103137+0000","last_undegraded":"2024-04-24T23:43:18.103137+0000","last_fullsized":"2024-04-24T23:43:18.103137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:52.857674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407550+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:42:56.407550+0000","last_peered":"2024-04-24T23:42:56.407550+0000","last_clean":"2024-04-24T23:42:56.407550+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:42:56.407550+0000","last_undegraded":"2024-04-24T23:42:56.407550+0000","last_fullsized":"2024-04-24T23:42:56.407550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406557+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:42:56.406557+0000","last_peered":"2024-04-24T23:42:56.406557+0000","last_clean":"2024-04-24T23:42:56.406557+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:42:56.406557+0000","last_undegraded":"2024-04-24T23:42:56.406557+0000","last_fullsized":"2024-04-24T23:42:56.406557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.100779+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:43:18.100779+0000","last_peered":"2024-04-24T23:43:18.100779+0000","last_clean":"2024-04-24T23:43:18.100779+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:43:18.100779+0000","last_undegraded":"2024-04-24T23:43:18.100779+0000","last_fullsized":"2024-04-24T23:43:18.100779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406977+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:42:56.406977+0000","last_peered":"2024-04-24T23:42:56.406977+0000","last_clean":"2024-04-24T23:42:56.406977+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:42:56.406977+0000","last_undegraded":"2024-04-24T23:42:56.406977+0000","last_fullsized":"2024-04-24T23:42:56.406977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314150+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:42:56.314150+0000","last_peered":"2024-04-24T23:42:56.314150+0000","last_clean":"2024-04-24T23:42:56.314150+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:42:56.314150+0000","last_undegraded":"2024-04-24T23:42:56.314150+0000","last_fullsized":"2024-04-24T23:42:56.314150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075846+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:43:15.075846+0000","last_peered":"2024-04-24T23:43:15.075846+0000","last_clean":"2024-04-24T23:43:15.075846+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:43:15.075846+0000","last_undegraded":"2024-04-24T23:43:15.075846+0000","last_fullsized":"2024-04-24T23:43:15.075846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.363982+0000","last_change":"2024-04-24T23:42:51.065658+0000","last_active":"2024-04-24T23:42:56.363982+0000","last_peered":"2024-04-24T23:42:56.363982+0000","last_clean":"2024-04-24T23:42:56.363982+0000","last_became_active":"2024-04-24T23:42:51.065295+0000","last_became_peered":"2024-04-24T23:42:51.065295+0000","last_unstale":"2024-04-24T23:42:56.363982+0000","last_undegraded":"2024-04-24T23:42:56.363982+0000","last_fullsized":"2024-04-24T23:42:56.363982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:17.536269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313996+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:42:56.313996+0000","last_peered":"2024-04-24T23:42:56.313996+0000","last_clean":"2024-04-24T23:42:56.313996+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:42:56.313996+0000","last_undegraded":"2024-04-24T23:42:56.313996+0000","last_fullsized":"2024-04-24T23:42:56.313996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407028+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:42:56.407028+0000","last_peered":"2024-04-24T23:42:56.407028+0000","last_clean":"2024-04-24T23:42:56.407028+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:42:56.407028+0000","last_undegraded":"2024-04-24T23:42:56.407028+0000","last_fullsized":"2024-04-24T23:42:56.407028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406962+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:42:56.406962+0000","last_peered":"2024-04-24T23:42:56.406962+0000","last_clean":"2024-04-24T23:42:56.406962+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:42:56.406962+0000","last_undegraded":"2024-04-24T23:42:56.406962+0000","last_fullsized":"2024-04-24T23:42:56.406962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406853+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:42:56.406853+0000","last_peered":"2024-04-24T23:42:56.406853+0000","last_clean":"2024-04-24T23:42:56.406853+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:42:56.406853+0000","last_undegraded":"2024-04-24T23:42:56.406853+0000","last_fullsized":"2024-04-24T23:42:56.406853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083897+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:43:16.083897+0000","last_peered":"2024-04-24T23:43:16.083897+0000","last_clean":"2024-04-24T23:43:16.083897+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:43:16.083897+0000","last_undegraded":"2024-04-24T23:43:16.083897+0000","last_fullsized":"2024-04-24T23:43:16.083897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406099+0000","last_change":"2024-04-24T23:42:51.073723+0000","last_active":"2024-04-24T23:42:56.406099+0000","last_peered":"2024-04-24T23:42:56.406099+0000","last_clean":"2024-04-24T23:42:56.406099+0000","last_became_active":"2024-04-24T23:42:51.073323+0000","last_became_peered":"2024-04-24T23:42:51.073323+0000","last_unstale":"2024-04-24T23:42:56.406099+0000","last_undegraded":"2024-04-24T23:42:56.406099+0000","last_fullsized":"2024-04-24T23:42:56.406099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:12.574032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079849+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:42:57.079849+0000","last_peered":"2024-04-24T23:42:57.079849+0000","last_clean":"2024-04-24T23:42:57.079849+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:42:57.079849+0000","last_undegraded":"2024-04-24T23:42:57.079849+0000","last_fullsized":"2024-04-24T23:42:57.079849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075584+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:43:15.075584+0000","last_peered":"2024-04-24T23:43:15.075584+0000","last_clean":"2024-04-24T23:43:15.075584+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:43:15.075584+0000","last_undegraded":"2024-04-24T23:43:15.075584+0000","last_fullsized":"2024-04-24T23:43:15.075584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315177+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:42:56.315177+0000","last_peered":"2024-04-24T23:42:56.315177+0000","last_clean":"2024-04-24T23:42:56.315177+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:42:56.315177+0000","last_undegraded":"2024-04-24T23:42:56.315177+0000","last_fullsized":"2024-04-24T23:42:56.315177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406559+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:42:56.406559+0000","last_peered":"2024-04-24T23:42:56.406559+0000","last_clean":"2024-04-24T23:42:56.406559+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:42:56.406559+0000","last_undegraded":"2024-04-24T23:42:56.406559+0000","last_fullsized":"2024-04-24T23:42:56.406559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.073446+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:43:15.073446+0000","last_peered":"2024-04-24T23:43:15.073446+0000","last_clean":"2024-04-24T23:43:15.073446+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:43:15.073446+0000","last_undegraded":"2024-04-24T23:43:15.073446+0000","last_fullsized":"2024-04-24T23:43:15.073446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406787+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:42:56.406787+0000","last_peered":"2024-04-24T23:42:56.406787+0000","last_clean":"2024-04-24T23:42:56.406787+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:42:56.406787+0000","last_undegraded":"2024-04-24T23:42:56.406787+0000","last_fullsized":"2024-04-24T23:42:56.406787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315124+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:42:56.315124+0000","last_peered":"2024-04-24T23:42:56.315124+0000","last_clean":"2024-04-24T23:42:56.315124+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:42:56.315124+0000","last_undegraded":"2024-04-24T23:42:56.315124+0000","last_fullsized":"2024-04-24T23:42:56.315124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.100837+0000","last_change":"2024-04-24T23:42:50.070204+0000","last_active":"2024-04-24T23:43:18.100837+0000","last_peered":"2024-04-24T23:43:18.100837+0000","last_clean":"2024-04-24T23:43:18.100837+0000","last_became_active":"2024-04-24T23:42:50.069708+0000","last_became_peered":"2024-04-24T23:42:50.069708+0000","last_unstale":"2024-04-24T23:43:18.100837+0000","last_undegraded":"2024-04-24T23:43:18.100837+0000","last_fullsized":"2024-04-24T23:43:18.100837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:55.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079542+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:42:57.079542+0000","last_peered":"2024-04-24T23:42:57.079542+0000","last_clean":"2024-04-24T23:42:57.079542+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:42:57.079542+0000","last_undegraded":"2024-04-24T23:42:57.079542+0000","last_fullsized":"2024-04-24T23:42:57.079542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407620+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:42:56.407620+0000","last_peered":"2024-04-24T23:42:56.407620+0000","last_clean":"2024-04-24T23:42:56.407620+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:42:56.407620+0000","last_undegraded":"2024-04-24T23:42:56.407620+0000","last_fullsized":"2024-04-24T23:42:56.407620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075307+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:43:15.075307+0000","last_peered":"2024-04-24T23:43:15.075307+0000","last_clean":"2024-04-24T23:43:15.075307+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:43:15.075307+0000","last_undegraded":"2024-04-24T23:43:15.075307+0000","last_fullsized":"2024-04-24T23:43:15.075307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407646+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:42:56.407646+0000","last_peered":"2024-04-24T23:42:56.407646+0000","last_clean":"2024-04-24T23:42:56.407646+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:42:56.407646+0000","last_undegraded":"2024-04-24T23:42:56.407646+0000","last_fullsized":"2024-04-24T23:42:56.407646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315179+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:42:56.315179+0000","last_peered":"2024-04-24T23:42:56.315179+0000","last_clean":"2024-04-24T23:42:56.315179+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:42:56.315179+0000","last_undegraded":"2024-04-24T23:42:56.315179+0000","last_fullsized":"2024-04-24T23:42:56.315179+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.074404+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:43:15.074404+0000","last_peered":"2024-04-24T23:43:15.074404+0000","last_clean":"2024-04-24T23:43:15.074404+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:43:15.074404+0000","last_undegraded":"2024-04-24T23:43:15.074404+0000","last_fullsized":"2024-04-24T23:43:15.074404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364279+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:42:56.364279+0000","last_peered":"2024-04-24T23:42:56.364279+0000","last_clean":"2024-04-24T23:42:56.364279+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:42:56.364279+0000","last_undegraded":"2024-04-24T23:42:56.364279+0000","last_fullsized":"2024-04-24T23:42:56.364279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407363+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:42:56.407363+0000","last_peered":"2024-04-24T23:42:56.407363+0000","last_clean":"2024-04-24T23:42:56.407363+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:42:56.407363+0000","last_undegraded":"2024-04-24T23:42:56.407363+0000","last_fullsized":"2024-04-24T23:42:56.407363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.100075+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:43:19.100075+0000","last_peered":"2024-04-24T23:43:19.100075+0000","last_clean":"2024-04-24T23:43:19.100075+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:43:19.100075+0000","last_undegraded":"2024-04-24T23:43:19.100075+0000","last_fullsized":"2024-04-24T23:43:19.100075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406365+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:42:56.406365+0000","last_peered":"2024-04-24T23:42:56.406365+0000","last_clean":"2024-04-24T23:42:56.406365+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:42:56.406365+0000","last_undegraded":"2024-04-24T23:42:56.406365+0000","last_fullsized":"2024-04-24T23:42:56.406365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406476+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:42:56.406476+0000","last_peered":"2024-04-24T23:42:56.406476+0000","last_clean":"2024-04-24T23:42:56.406476+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:42:56.406476+0000","last_undegraded":"2024-04-24T23:42:56.406476+0000","last_fullsized":"2024-04-24T23:42:56.406476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.075778+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:43:15.075778+0000","last_peered":"2024-04-24T23:43:15.075778+0000","last_clean":"2024-04-24T23:43:15.075778+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:43:15.075778+0000","last_undegraded":"2024-04-24T23:43:15.075778+0000","last_fullsized":"2024-04-24T23:43:15.075778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314064+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:42:56.314064+0000","last_peered":"2024-04-24T23:42:56.314064+0000","last_clean":"2024-04-24T23:42:56.314064+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:42:56.314064+0000","last_undegraded":"2024-04-24T23:42:56.314064+0000","last_fullsized":"2024-04-24T23:42:56.314064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315673+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:42:56.315673+0000","last_peered":"2024-04-24T23:42:56.315673+0000","last_clean":"2024-04-24T23:42:56.315673+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:42:56.315673+0000","last_undegraded":"2024-04-24T23:42:56.315673+0000","last_fullsized":"2024-04-24T23:42:56.315673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.102764+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:43:19.102764+0000","last_peered":"2024-04-24T23:43:19.102764+0000","last_clean":"2024-04-24T23:43:19.102764+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:43:19.102764+0000","last_undegraded":"2024-04-24T23:43:19.102764+0000","last_fullsized":"2024-04-24T23:43:19.102764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314048+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:42:56.314048+0000","last_peered":"2024-04-24T23:42:56.314048+0000","last_clean":"2024-04-24T23:42:56.314048+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:42:56.314048+0000","last_undegraded":"2024-04-24T23:42:56.314048+0000","last_fullsized":"2024-04-24T23:42:56.314048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406838+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:42:56.406838+0000","last_peered":"2024-04-24T23:42:56.406838+0000","last_clean":"2024-04-24T23:42:56.406838+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:42:56.406838+0000","last_undegraded":"2024-04-24T23:42:56.406838+0000","last_fullsized":"2024-04-24T23:42:56.406838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.091068+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:43:17.091068+0000","last_peered":"2024-04-24T23:43:17.091068+0000","last_clean":"2024-04-24T23:43:17.091068+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:43:17.091068+0000","last_undegraded":"2024-04-24T23:43:17.091068+0000","last_fullsized":"2024-04-24T23:43:17.091068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407479+0000","last_change":"2024-04-24T23:42:51.076145+0000","last_active":"2024-04-24T23:42:56.407479+0000","last_peered":"2024-04-24T23:42:56.407479+0000","last_clean":"2024-04-24T23:42:56.407479+0000","last_became_active":"2024-04-24T23:42:51.073075+0000","last_became_peered":"2024-04-24T23:42:51.073075+0000","last_unstale":"2024-04-24T23:42:56.407479+0000","last_undegraded":"2024-04-24T23:42:56.407479+0000","last_fullsized":"2024-04-24T23:42:56.407479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:04.519183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407375+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:42:56.407375+0000","last_peered":"2024-04-24T23:42:56.407375+0000","last_clean":"2024-04-24T23:42:56.407375+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:42:56.407375+0000","last_undegraded":"2024-04-24T23:42:56.407375+0000","last_fullsized":"2024-04-24T23:42:56.407375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364318+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:42:56.364318+0000","last_peered":"2024-04-24T23:42:56.364318+0000","last_clean":"2024-04-24T23:42:56.364318+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:42:56.364318+0000","last_undegraded":"2024-04-24T23:42:56.364318+0000","last_fullsized":"2024-04-24T23:42:56.364318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406627+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:42:56.406627+0000","last_peered":"2024-04-24T23:42:56.406627+0000","last_clean":"2024-04-24T23:42:56.406627+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:42:56.406627+0000","last_undegraded":"2024-04-24T23:42:56.406627+0000","last_fullsized":"2024-04-24T23:42:56.406627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.364128+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:42:56.364128+0000","last_peered":"2024-04-24T23:42:56.364128+0000","last_clean":"2024-04-24T23:42:56.364128+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:42:56.364128+0000","last_undegraded":"2024-04-24T23:42:56.364128+0000","last_fullsized":"2024-04-24T23:42:56.364128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315408+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:42:56.315408+0000","last_peered":"2024-04-24T23:42:56.315408+0000","last_clean":"2024-04-24T23:42:56.315408+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:42:56.315408+0000","last_undegraded":"2024-04-24T23:42:56.315408+0000","last_fullsized":"2024-04-24T23:42:56.315408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407196+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:42:56.407196+0000","last_peered":"2024-04-24T23:42:56.407196+0000","last_clean":"2024-04-24T23:42:56.407196+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:42:56.407196+0000","last_undegraded":"2024-04-24T23:42:56.407196+0000","last_fullsized":"2024-04-24T23:42:56.407196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407024+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:42:56.407024+0000","last_peered":"2024-04-24T23:42:56.407024+0000","last_clean":"2024-04-24T23:42:56.407024+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:42:56.407024+0000","last_undegraded":"2024-04-24T23:42:56.407024+0000","last_fullsized":"2024-04-24T23:42:56.407024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.101679+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:43:19.101679+0000","last_peered":"2024-04-24T23:43:19.101679+0000","last_clean":"2024-04-24T23:43:19.101679+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:43:19.101679+0000","last_undegraded":"2024-04-24T23:43:19.101679+0000","last_fullsized":"2024-04-24T23:43:19.101679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407472+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:42:56.407472+0000","last_peered":"2024-04-24T23:42:56.407472+0000","last_clean":"2024-04-24T23:42:56.407472+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:42:56.407472+0000","last_undegraded":"2024-04-24T23:42:56.407472+0000","last_fullsized":"2024-04-24T23:42:56.407472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406473+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:42:56.406473+0000","last_peered":"2024-04-24T23:42:56.406473+0000","last_clean":"2024-04-24T23:42:56.406473+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:42:56.406473+0000","last_undegraded":"2024-04-24T23:42:56.406473+0000","last_fullsized":"2024-04-24T23:42:56.406473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406521+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:42:56.406521+0000","last_peered":"2024-04-24T23:42:56.406521+0000","last_clean":"2024-04-24T23:42:56.406521+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:42:56.406521+0000","last_undegraded":"2024-04-24T23:42:56.406521+0000","last_fullsized":"2024-04-24T23:42:56.406521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083222+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:43:16.083222+0000","last_peered":"2024-04-24T23:43:16.083222+0000","last_clean":"2024-04-24T23:43:16.083222+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:43:16.083222+0000","last_undegraded":"2024-04-24T23:43:16.083222+0000","last_fullsized":"2024-04-24T23:43:16.083222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314092+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:42:56.314092+0000","last_peered":"2024-04-24T23:42:56.314092+0000","last_clean":"2024-04-24T23:42:56.314092+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:42:56.314092+0000","last_undegraded":"2024-04-24T23:42:56.314092+0000","last_fullsized":"2024-04-24T23:42:56.314092+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406005+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:42:56.406005+0000","last_peered":"2024-04-24T23:42:56.406005+0000","last_clean":"2024-04-24T23:42:56.406005+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:42:56.406005+0000","last_undegraded":"2024-04-24T23:42:56.406005+0000","last_fullsized":"2024-04-24T23:42:56.406005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.099769+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:43:19.099769+0000","last_peered":"2024-04-24T23:43:19.099769+0000","last_clean":"2024-04-24T23:43:19.099769+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:43:19.099769+0000","last_undegraded":"2024-04-24T23:43:19.099769+0000","last_fullsized":"2024-04-24T23:43:19.099769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313570+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:42:56.313570+0000","last_peered":"2024-04-24T23:42:56.313570+0000","last_clean":"2024-04-24T23:42:56.313570+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:42:56.313570+0000","last_undegraded":"2024-04-24T23:42:56.313570+0000","last_fullsized":"2024-04-24T23:42:56.313570+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407419+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:42:56.407419+0000","last_peered":"2024-04-24T23:42:56.407419+0000","last_clean":"2024-04-24T23:42:56.407419+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:42:56.407419+0000","last_undegraded":"2024-04-24T23:42:56.407419+0000","last_fullsized":"2024-04-24T23:42:56.407419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.405959+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:42:56.405959+0000","last_peered":"2024-04-24T23:42:56.405959+0000","last_clean":"2024-04-24T23:42:56.405959+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:42:56.405959+0000","last_undegraded":"2024-04-24T23:42:56.405959+0000","last_fullsized":"2024-04-24T23:42:56.405959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083796+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:43:16.083796+0000","last_peered":"2024-04-24T23:43:16.083796+0000","last_clean":"2024-04-24T23:43:16.083796+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:43:16.083796+0000","last_undegraded":"2024-04-24T23:43:16.083796+0000","last_fullsized":"2024-04-24T23:43:16.083796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406453+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:42:56.406453+0000","last_peered":"2024-04-24T23:42:56.406453+0000","last_clean":"2024-04-24T23:42:56.406453+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:42:56.406453+0000","last_undegraded":"2024-04-24T23:42:56.406453+0000","last_fullsized":"2024-04-24T23:42:56.406453+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.091142+0000","last_change":"2024-04-24T23:42:50.074409+0000","last_active":"2024-04-24T23:43:17.091142+0000","last_peered":"2024-04-24T23:43:17.091142+0000","last_clean":"2024-04-24T23:43:17.091142+0000","last_became_active":"2024-04-24T23:42:50.072784+0000","last_became_peered":"2024-04-24T23:42:50.072784+0000","last_unstale":"2024-04-24T23:43:17.091142+0000","last_undegraded":"2024-04-24T23:43:17.091142+0000","last_fullsized":"2024-04-24T23:43:17.091142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:08.625302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407062+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:42:56.407062+0000","last_peered":"2024-04-24T23:42:56.407062+0000","last_clean":"2024-04-24T23:42:56.407062+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:42:56.407062+0000","last_undegraded":"2024-04-24T23:42:56.407062+0000","last_fullsized":"2024-04-24T23:42:56.407062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406056+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:42:56.406056+0000","last_peered":"2024-04-24T23:42:56.406056+0000","last_clean":"2024-04-24T23:42:56.406056+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:42:56.406056+0000","last_undegraded":"2024-04-24T23:42:56.406056+0000","last_fullsized":"2024-04-24T23:42:56.406056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:16.083767+0000","last_change":"2024-04-24T23:42:50.068569+0000","last_active":"2024-04-24T23:43:16.083767+0000","last_peered":"2024-04-24T23:43:16.083767+0000","last_clean":"2024-04-24T23:43:16.083767+0000","last_became_active":"2024-04-24T23:42:50.068078+0000","last_became_peered":"2024-04-24T23:42:50.068078+0000","last_unstale":"2024-04-24T23:43:16.083767+0000","last_undegraded":"2024-04-24T23:43:16.083767+0000","last_fullsized":"2024-04-24T23:43:16.083767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:48:15.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406635+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:42:56.406635+0000","last_peered":"2024-04-24T23:42:56.406635+0000","last_clean":"2024-04-24T23:42:56.406635+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:42:56.406635+0000","last_undegraded":"2024-04-24T23:42:56.406635+0000","last_fullsized":"2024-04-24T23:42:56.406635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313761+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:42:56.313761+0000","last_peered":"2024-04-24T23:42:56.313761+0000","last_clean":"2024-04-24T23:42:56.313761+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:42:56.313761+0000","last_undegraded":"2024-04-24T23:42:56.313761+0000","last_fullsized":"2024-04-24T23:42:56.313761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.102825+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:43:18.102825+0000","last_peered":"2024-04-24T23:43:18.102825+0000","last_clean":"2024-04-24T23:43:18.102825+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:43:18.102825+0000","last_undegraded":"2024-04-24T23:43:18.102825+0000","last_fullsized":"2024-04-24T23:43:18.102825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079788+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:42:57.079788+0000","last_peered":"2024-04-24T23:42:57.079788+0000","last_clean":"2024-04-24T23:42:57.079788+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:42:57.079788+0000","last_undegraded":"2024-04-24T23:42:57.079788+0000","last_fullsized":"2024-04-24T23:42:57.079788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407467+0000","last_change":"2024-04-24T23:42:55.322284+0000","last_active":"2024-04-24T23:42:56.407467+0000","last_peered":"2024-04-24T23:42:56.407467+0000","last_clean":"2024-04-24T23:42:56.407467+0000","last_became_active":"2024-04-24T23:42:55.321228+0000","last_became_peered":"2024-04-24T23:42:55.321228+0000","last_unstale":"2024-04-24T23:42:56.407467+0000","last_undegraded":"2024-04-24T23:42:56.407467+0000","last_fullsized":"2024-04-24T23:42:56.407467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:20.864377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.074256+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:43:15.074256+0000","last_peered":"2024-04-24T23:43:15.074256+0000","last_clean":"2024-04-24T23:43:15.074256+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:43:15.074256+0000","last_undegraded":"2024-04-24T23:43:15.074256+0000","last_fullsized":"2024-04-24T23:43:15.074256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.363820+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:42:56.363820+0000","last_peered":"2024-04-24T23:42:56.363820+0000","last_clean":"2024-04-24T23:42:56.363820+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:42:56.363820+0000","last_undegraded":"2024-04-24T23:42:56.363820+0000","last_fullsized":"2024-04-24T23:42:56.363820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.315779+0000","last_change":"2024-04-24T23:42:55.324683+0000","last_active":"2024-04-24T23:42:56.315779+0000","last_peered":"2024-04-24T23:42:56.315779+0000","last_clean":"2024-04-24T23:42:56.315779+0000","last_became_active":"2024-04-24T23:42:55.324311+0000","last_became_peered":"2024-04-24T23:42:55.324311+0000","last_unstale":"2024-04-24T23:42:56.315779+0000","last_undegraded":"2024-04-24T23:42:56.315779+0000","last_fullsized":"2024-04-24T23:42:56.315779+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:17.659811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.102158+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:43:18.102158+0000","last_peered":"2024-04-24T23:43:18.102158+0000","last_clean":"2024-04-24T23:43:18.102158+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:43:18.102158+0000","last_undegraded":"2024-04-24T23:43:18.102158+0000","last_fullsized":"2024-04-24T23:43:18.102158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406114+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:42:56.406114+0000","last_peered":"2024-04-24T23:42:56.406114+0000","last_clean":"2024-04-24T23:42:56.406114+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:42:56.406114+0000","last_undegraded":"2024-04-24T23:42:56.406114+0000","last_fullsized":"2024-04-24T23:42:56.406114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:59.181180+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:42:59.181180+0000","last_peered":"2024-04-24T23:42:59.181180+0000","last_clean":"2024-04-24T23:42:59.181180+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:42:59.181180+0000","last_undegraded":"2024-04-24T23:42:59.181180+0000","last_fullsized":"2024-04-24T23:42:59.181180+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:17.090859+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:43:17.090859+0000","last_peered":"2024-04-24T23:43:17.090859+0000","last_clean":"2024-04-24T23:43:17.090859+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:43:17.090859+0000","last_undegraded":"2024-04-24T23:43:17.090859+0000","last_fullsized":"2024-04-24T23:43:17.090859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406511+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:42:56.406511+0000","last_peered":"2024-04-24T23:42:56.406511+0000","last_clean":"2024-04-24T23:42:56.406511+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:42:56.406511+0000","last_undegraded":"2024-04-24T23:42:56.406511+0000","last_fullsized":"2024-04-24T23:42:56.406511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407495+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:42:56.407495+0000","last_peered":"2024-04-24T23:42:56.407495+0000","last_clean":"2024-04-24T23:42:56.407495+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:42:56.407495+0000","last_undegraded":"2024-04-24T23:42:56.407495+0000","last_fullsized":"2024-04-24T23:42:56.407495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:15.073685+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:43:15.073685+0000","last_peered":"2024-04-24T23:43:15.073685+0000","last_clean":"2024-04-24T23:43:15.073685+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:43:15.073685+0000","last_undegraded":"2024-04-24T23:43:15.073685+0000","last_fullsized":"2024-04-24T23:43:15.073685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313952+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:42:56.313952+0000","last_peered":"2024-04-24T23:42:56.313952+0000","last_clean":"2024-04-24T23:42:56.313952+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:42:56.313952+0000","last_undegraded":"2024-04-24T23:42:56.313952+0000","last_fullsized":"2024-04-24T23:42:56.313952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407041+0000","last_change":"2024-04-24T23:42:55.319168+0000","last_active":"2024-04-24T23:42:56.407041+0000","last_peered":"2024-04-24T23:42:56.407041+0000","last_clean":"2024-04-24T23:42:56.407041+0000","last_became_active":"2024-04-24T23:42:55.318752+0000","last_became_peered":"2024-04-24T23:42:55.318752+0000","last_unstale":"2024-04-24T23:42:56.407041+0000","last_undegraded":"2024-04-24T23:42:56.407041+0000","last_fullsized":"2024-04-24T23:42:56.407041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:14.923700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:19.102925+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:43:19.102925+0000","last_peered":"2024-04-24T23:43:19.102925+0000","last_clean":"2024-04-24T23:43:19.102925+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:43:19.102925+0000","last_undegraded":"2024-04-24T23:43:19.102925+0000","last_fullsized":"2024-04-24T23:43:19.102925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314322+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:42:56.314322+0000","last_peered":"2024-04-24T23:42:56.314322+0000","last_clean":"2024-04-24T23:42:56.314322+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:42:56.314322+0000","last_undegraded":"2024-04-24T23:42:56.314322+0000","last_fullsized":"2024-04-24T23:42:56.314322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079486+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:42:57.079486+0000","last_peered":"2024-04-24T23:42:57.079486+0000","last_clean":"2024-04-24T23:42:57.079486+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:42:57.079486+0000","last_undegraded":"2024-04-24T23:42:57.079486+0000","last_fullsized":"2024-04-24T23:42:57.079486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407086+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:42:56.407086+0000","last_peered":"2024-04-24T23:42:56.407086+0000","last_clean":"2024-04-24T23:42:56.407086+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:42:56.407086+0000","last_undegraded":"2024-04-24T23:42:56.407086+0000","last_fullsized":"2024-04-24T23:42:56.407086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314247+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:42:56.314247+0000","last_peered":"2024-04-24T23:42:56.314247+0000","last_clean":"2024-04-24T23:42:56.314247+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:42:56.314247+0000","last_undegraded":"2024-04-24T23:42:56.314247+0000","last_fullsized":"2024-04-24T23:42:56.314247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406216+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:42:56.406216+0000","last_peered":"2024-04-24T23:42:56.406216+0000","last_clean":"2024-04-24T23:42:56.406216+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:42:56.406216+0000","last_undegraded":"2024-04-24T23:42:56.406216+0000","last_fullsized":"2024-04-24T23:42:56.406216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:43:18.099083+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:43:18.099083+0000","last_peered":"2024-04-24T23:43:18.099083+0000","last_clean":"2024-04-24T23:43:18.099083+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:43:18.099083+0000","last_undegraded":"2024-04-24T23:43:18.099083+0000","last_fullsized":"2024-04-24T23:43:18.099083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314311+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:42:56.314311+0000","last_peered":"2024-04-24T23:42:56.314311+0000","last_clean":"2024-04-24T23:42:56.314311+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:42:56.314311+0000","last_undegraded":"2024-04-24T23:42:56.314311+0000","last_fullsized":"2024-04-24T23:42:56.314311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406181+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:42:56.406181+0000","last_peered":"2024-04-24T23:42:56.406181+0000","last_clean":"2024-04-24T23:42:56.406181+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:42:56.406181+0000","last_undegraded":"2024-04-24T23:42:56.406181+0000","last_fullsized":"2024-04-24T23:42:56.406181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313579+0000","last_change":"2024-04-24T23:42:51.081821+0000","last_active":"2024-04-24T23:42:56.313579+0000","last_peered":"2024-04-24T23:42:56.313579+0000","last_clean":"2024-04-24T23:42:56.313579+0000","last_became_active":"2024-04-24T23:42:51.081246+0000","last_became_peered":"2024-04-24T23:42:51.081246+0000","last_unstale":"2024-04-24T23:42:56.313579+0000","last_undegraded":"2024-04-24T23:42:56.313579+0000","last_fullsized":"2024-04-24T23:42:56.313579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:24.375428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.313501+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:42:56.313501+0000","last_peered":"2024-04-24T23:42:56.313501+0000","last_clean":"2024-04-24T23:42:56.313501+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:42:56.313501+0000","last_undegraded":"2024-04-24T23:42:56.313501+0000","last_fullsized":"2024-04-24T23:42:56.313501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:57.079503+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:42:57.079503+0000","last_peered":"2024-04-24T23:42:57.079503+0000","last_clean":"2024-04-24T23:42:57.079503+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:42:57.079503+0000","last_undegraded":"2024-04-24T23:42:57.079503+0000","last_fullsized":"2024-04-24T23:42:57.079503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406976+0000","last_change":"2024-04-24T23:42:51.070156+0000","last_active":"2024-04-24T23:42:56.406976+0000","last_peered":"2024-04-24T23:42:56.406976+0000","last_clean":"2024-04-24T23:42:56.406976+0000","last_became_active":"2024-04-24T23:42:51.069842+0000","last_became_peered":"2024-04-24T23:42:51.069842+0000","last_unstale":"2024-04-24T23:42:56.406976+0000","last_undegraded":"2024-04-24T23:42:56.406976+0000","last_fullsized":"2024-04-24T23:42:56.406976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:06.133128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406728+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:42:56.406728+0000","last_peered":"2024-04-24T23:42:56.406728+0000","last_clean":"2024-04-24T23:42:56.406728+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:42:56.406728+0000","last_undegraded":"2024-04-24T23:42:56.406728+0000","last_fullsized":"2024-04-24T23:42:56.406728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.314234+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:42:56.314234+0000","last_peered":"2024-04-24T23:42:56.314234+0000","last_clean":"2024-04-24T23:42:56.314234+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:42:56.314234+0000","last_undegraded":"2024-04-24T23:42:56.314234+0000","last_fullsized":"2024-04-24T23:42:56.314234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406728+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:42:56.406728+0000","last_peered":"2024-04-24T23:42:56.406728+0000","last_clean":"2024-04-24T23:42:56.406728+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:42:56.406728+0000","last_undegraded":"2024-04-24T23:42:56.406728+0000","last_fullsized":"2024-04-24T23:42:56.406728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406407+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:42:56.406407+0000","last_peered":"2024-04-24T23:42:56.406407+0000","last_clean":"2024-04-24T23:42:56.406407+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:42:56.406407+0000","last_undegraded":"2024-04-24T23:42:56.406407+0000","last_fullsized":"2024-04-24T23:42:56.406407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406711+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:42:56.406711+0000","last_peered":"2024-04-24T23:42:56.406711+0000","last_clean":"2024-04-24T23:42:56.406711+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:42:56.406711+0000","last_undegraded":"2024-04-24T23:42:56.406711+0000","last_fullsized":"2024-04-24T23:42:56.406711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.407357+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:42:56.407357+0000","last_peered":"2024-04-24T23:42:56.407357+0000","last_clean":"2024-04-24T23:42:56.407357+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:42:56.407357+0000","last_undegraded":"2024-04-24T23:42:56.407357+0000","last_fullsized":"2024-04-24T23:42:56.407357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406237+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:42:56.406237+0000","last_peered":"2024-04-24T23:42:56.406237+0000","last_clean":"2024-04-24T23:42:56.406237+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:42:56.406237+0000","last_undegraded":"2024-04-24T23:42:56.406237+0000","last_fullsized":"2024-04-24T23:42:56.406237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-24T23:42:56.406432+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:42:56.406432+0000","last_peered":"2024-04-24T23:42:56.406432+0000","last_clean":"2024-04-24T23:42:56.406432+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:42:56.406432+0000","last_undegraded":"2024-04-24T23:42:56.406432+0000","last_fullsized":"2024-04-24T23:42:56.406432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10475,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":185,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":64673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":478407,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478715,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":1079308,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477937,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479281,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":502328,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477623,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478306,"data_compressed":14677,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1069821,"data_compressed":50863,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:43:20.802 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T23:43:20.803 INFO:tasks.cephadm:Setting up client nodes... 2024-04-24T23:43:20.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-24T23:43:21.072 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:21.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:21 smithi107 ceph-mon[29118]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:21.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:21 smithi107 ceph-mon[29118]: pgmap v305: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-04-24T23:43:21.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:21 smithi107 ceph-mon[29118]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T23:43:21.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:21 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1988015121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:21 smithi195 ceph-mon[33019]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:21 smithi195 ceph-mon[33019]: pgmap v305: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-04-24T23:43:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:21 smithi195 ceph-mon[33019]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T23:43:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:21 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1988015121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:21 smithi170 ceph-mon[32714]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:43:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:21 smithi170 ceph-mon[32714]: pgmap v305: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-04-24T23:43:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:21 smithi170 ceph-mon[32714]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T23:43:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:21 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1988015121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:22.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:22 smithi107 ceph-mon[29118]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:22 smithi107 ceph-mon[29118]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:22 smithi195 ceph-mon[33019]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:22 smithi195 ceph-mon[33019]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:22 smithi170 ceph-mon[32714]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:22 smithi170 ceph-mon[32714]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:43:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- 172.21.15.107:0/371961040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102d70 msgr2=0x7fc9f0109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/371961040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102d70 0x7fc9f0109620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800b0d0 tx=0x7fc9e800b3e0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- 172.21.15.107:0/371961040 shutdown_connections 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/371961040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102d70 0x7fc9f0109620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/371961040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f01011d0 0x7fc9f01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/371961040 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 0x7fc9f0102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- 172.21.15.107:0/371961040 >> 172.21.15.107:0/371961040 conn(0x7fc9f00fc790 msgr2=0x7fc9f00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- 172.21.15.107:0/371961040 shutdown_connections 2024-04-24T23:43:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- 172.21.15.107:0/371961040 wait complete. 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 Processor -- start 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.875+0000 7fc9f49f9700 1 -- start start 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 0x7fc9f019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f0102d70 0x7fc9f01a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f010cdd0 con 0x7fc9f01011d0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f010cc50 con 0x7fc9f0102d70 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f010cf50 con 0x7fc9f01023d0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42628/0 (socket says 172.21.15.107:42628) 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 -- 172.21.15.107:0/1969556953 learned_addr learned my addr 172.21.15.107:0/1969556953 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 0x7fc9f019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9eed9d700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f0102d70 0x7fc9f01a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 -- 172.21.15.107:0/1969556953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 msgr2=0x7fc9f019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 0x7fc9f019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 -- 172.21.15.107:0/1969556953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f0102d70 msgr2=0x7fc9f01a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f0102d70 0x7fc9f01a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 -- 172.21.15.107:0/1969556953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f01a21f0 con 0x7fc9f01011d0 2024-04-24T23:43:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c970 tx=0x7fc9e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0004220 con 0x7fc9f01011d0 2024-04-24T23:43:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9e00043c0 con 0x7fc9f01011d0 2024-04-24T23:43:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f01a24e0 con 0x7fc9f01011d0 2024-04-24T23:43:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.876+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0003d40 con 0x7fc9f01011d0 2024-04-24T23:43:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.877+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f01a2a70 con 0x7fc9f01011d0 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.878+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f00507a0 con 0x7fc9f01011d0 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.878+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc9e00076d0 con 0x7fc9f01011d0 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.879+0000 7fc9df7fe700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 0x7fc9d80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.879+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 0x7fc9d80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.879+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc9e00cf1a0 con 0x7fc9f01011d0 2024-04-24T23:43:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.879+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 0x7fc9d80a52a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc9f006ba80 tx=0x7fc9e400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:22.883+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9e0014030 con 0x7fc9f01011d0 2024-04-24T23:43:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.168+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fc9f002db30 con 0x7fc9f01011d0 2024-04-24T23:43:23.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.170+0000 7fc9df7fe700 1 -- 172.21.15.107:0/1969556953 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fc9e009bd60 con 0x7fc9f01011d0 2024-04-24T23:43:23.170 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-04-24T23:43:23.170 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQAbmSlmA0YhChAAqzW5WldV1O7Zklkvx32qJw== 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 msgr2=0x7fc9d80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 0x7fc9d80a52a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc9f006ba80 tx=0x7fc9e400b3c0 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 msgr2=0x7fc9f006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c970 tx=0x7fc9e000cc80 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 shutdown_connections 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc9d80a2de0 0x7fc9d80a52a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9f0102d70 0x7fc9f01a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f01011d0 0x7fc9f006b500 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 --2- 172.21.15.107:0/1969556953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc9f01023d0 0x7fc9f019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 >> 172.21.15.107:0/1969556953 conn(0x7fc9f00fc790 msgr2=0x7fc9f0105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 shutdown_connections 2024-04-24T23:43:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:23.173+0000 7fc9f49f9700 1 -- 172.21.15.107:0/1969556953 wait complete. 2024-04-24T23:43:23.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:23 smithi107 ceph-mon[29118]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-24T23:43:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:23 smithi195 ceph-mon[33019]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-24T23:43:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:23 smithi170 ceph-mon[32714]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-24T23:43:24.059 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:43:24.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-24T23:43:24.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T23:43:24.096 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T23:43:24.096 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-24T23:43:24.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-24T23:43:24.372 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:24.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:24 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T23:43:24.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:24 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T23:43:24.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:24 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T23:43:24.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:24 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T23:43:24.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:24 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T23:43:24.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:24 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1969556953' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T23:43:25.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:25 smithi107 ceph-mon[29118]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-24T23:43:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:25 smithi195 ceph-mon[33019]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-24T23:43:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:25 smithi170 ceph-mon[32714]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-24T23:43:26.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 -- 172.21.15.107:0/1135729640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c102550 msgr2=0x7fa40c102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1135729640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c102550 0x7fa40c102930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0099c0 tx=0x7fa3fc009cd0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 -- 172.21.15.107:0/1135729640 shutdown_connections 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1135729640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c107540 0x7fa40c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1135729640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c102550 0x7fa40c102930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1135729640 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102f00 0x7fa40c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 -- 172.21.15.107:0/1135729640 >> 172.21.15.107:0/1135729640 conn(0x7fa40c0793f0 msgr2=0x7fa40c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 -- 172.21.15.107:0/1135729640 shutdown_connections 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.143+0000 7fa411ca5700 1 -- 172.21.15.107:0/1135729640 wait complete. 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 Processor -- start 2024-04-24T23:43:26.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 -- start start 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102550 0x7fa40c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c102f00 0x7fa40c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c1125a0 con 0x7fa40c107540 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c112420 con 0x7fa40c102f00 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa411ca5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c112720 con 0x7fa40c102550 2024-04-24T23:43:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.144+0000 7fa40bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42652/0 (socket says 172.21.15.107:42652) 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 -- 172.21.15.107:0/1580075327 learned_addr learned my addr 172.21.15.107:0/1580075327 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40affd700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c102f00 0x7fa40c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40b7fe700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102550 0x7fa40c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 -- 172.21.15.107:0/1580075327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102550 msgr2=0x7fa40c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102550 0x7fa40c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 -- 172.21.15.107:0/1580075327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c102f00 msgr2=0x7fa40c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c102f00 0x7fa40c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 -- 172.21.15.107:0/1580075327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40c1a6ed0 con 0x7fa40c107540 2024-04-24T23:43:26.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa40bfff700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa40400b970 tx=0x7fa40400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404019070 con 0x7fa40c107540 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40c1a71c0 con 0x7fa40c107540 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa40400ce10 con 0x7fa40c107540 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404007370 con 0x7fa40c107540 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.145+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40c1a7700 con 0x7fa40c107540 2024-04-24T23:43:26.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.146+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40c068f20 con 0x7fa40c107540 2024-04-24T23:43:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.147+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa40401c070 con 0x7fa40c107540 2024-04-24T23:43:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.148+0000 7fa408ff9700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 0x7fa3f40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.148+0000 7fa40b7fe700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 0x7fa3f40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.148+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4040ceb80 con 0x7fa40c107540 2024-04-24T23:43:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.148+0000 7fa40b7fe700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 0x7fa3f40a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009750 tx=0x7fa3fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:26.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.152+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa40409b850 con 0x7fa40c107540 2024-04-24T23:43:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.408+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa40c075a10 con 0x7fa40c107540 2024-04-24T23:43:26.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.412+0000 7fa408ff9700 1 -- 172.21.15.107:0/1580075327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263210 (secure 0 0 0) 0x7fa40409b150 con 0x7fa40c107540 2024-04-24T23:43:26.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 msgr2=0x7fa3f40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 0x7fa3f40a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009750 tx=0x7fa3fc005fd0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 msgr2=0x7fa40c074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa40400b970 tx=0x7fa40400bc80 comp rx=0 tx=0).stop 2024-04-24T23:43:26.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 shutdown_connections 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa3f40a2cb0 0x7fa3f40a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40c107540 0x7fa40c074fb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa40c102550 0x7fa40c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 --2- 172.21.15.107:0/1580075327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa40c102f00 0x7fa40c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 >> 172.21.15.107:0/1580075327 conn(0x7fa40c0793f0 msgr2=0x7fa40c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 shutdown_connections 2024-04-24T23:43:26.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:26.416+0000 7fa411ca5700 1 -- 172.21.15.107:0/1580075327 wait complete. 2024-04-24T23:43:27.156 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1159279165},{"type":"v1","addr":"172.21.15.107:6801","nonce":1159279165}]},"active_addr":"172.21.15.107:6801/1159279165","active_change":"2024-04-24T23:35:47.891537+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1579426836}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2857053143}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3291929000}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3569834870}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3607394468}]}]} 2024-04-24T23:43:27.165 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-24T23:43:27.166 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-24T23:43:27.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T23:43:27.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:27 smithi107 ceph-mon[29118]: pgmap v308: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 23 op/s 2024-04-24T23:43:27.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:27 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1580075327' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T23:43:27.413 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:27 smithi195 ceph-mon[33019]: pgmap v308: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 23 op/s 2024-04-24T23:43:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:27 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1580075327' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T23:43:27.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:27 smithi170 ceph-mon[32714]: pgmap v308: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 23 op/s 2024-04-24T23:43:27.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:27 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1580075327' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T23:43:29.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 -- 172.21.15.107:0/791418628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 msgr2=0x7f25c410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:29.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 --2- 172.21.15.107:0/791418628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c410dfe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00b0a0 tx=0x7f25bc00b3b0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 -- 172.21.15.107:0/791418628 shutdown_connections 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 --2- 172.21.15.107:0/791418628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c410dfe0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 --2- 172.21.15.107:0/791418628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 0x7f25c4107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 --2- 172.21.15.107:0/791418628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.197+0000 7f25c8b34700 1 -- 172.21.15.107:0/791418628 >> 172.21.15.107:0/791418628 conn(0x7f25c40780e0 msgr2=0x7f25c40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.198+0000 7f25c8b34700 1 -- 172.21.15.107:0/791418628 shutdown_connections 2024-04-24T23:43:29.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.198+0000 7f25c8b34700 1 -- 172.21.15.107:0/791418628 wait complete. 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 Processor -- start 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 -- start start 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 0x7f25c41a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:29.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4111510 con 0x7f25c4101af0 2024-04-24T23:43:29.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4111390 con 0x7f25c4101130 2024-04-24T23:43:29.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c8b34700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4111690 con 0x7f25c41071a0 2024-04-24T23:43:29.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:29.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43294/0 (socket says 172.21.15.107:43294) 2024-04-24T23:43:29.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 -- 172.21.15.107:0/3437355210 learned_addr learned my addr 172.21.15.107:0/3437355210 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c2d9d700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 0x7f25c41a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c259c700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 -- 172.21.15.107:0/3437355210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 msgr2=0x7f25c41a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 0x7f25c41a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 -- 172.21.15.107:0/3437355210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 msgr2=0x7f25c41998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c1d9b700 1 -- 172.21.15.107:0/3437355210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c41a4e90 con 0x7f25c4101af0 2024-04-24T23:43:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.199+0000 7f25c259c700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:29.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25c1d9b700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f25b800b530 tx=0x7f25b800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8015020 con 0x7f25c4101af0 2024-04-24T23:43:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b8011d10 con 0x7f25c4101af0 2024-04-24T23:43:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c41a5060 con 0x7f25c4101af0 2024-04-24T23:43:29.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b800f2e0 con 0x7f25c4101af0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.200+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c41a5570 con 0x7f25c4101af0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.201+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c4109710 con 0x7f25c4101af0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.202+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f25b8018020 con 0x7f25c4101af0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.202+0000 7f25b37fe700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 0x7f25ac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.202+0000 7f25c259c700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 0x7f25ac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.202+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f25b80ceb60 con 0x7f25c4101af0 2024-04-24T23:43:29.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.203+0000 7f25c259c700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 0x7f25ac0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25b40099d0 tx=0x7f25b4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:29.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.207+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25b801f020 con 0x7f25c4101af0 2024-04-24T23:43:29.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:29 smithi107 ceph-mon[29118]: pgmap v309: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.8 KiB/s wr, 23 op/s 2024-04-24T23:43:29.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.404+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25c406a4b0 con 0x7f25c4101af0 2024-04-24T23:43:29.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.406+0000 7f25b37fe700 1 -- 172.21.15.107:0/3437355210 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f25b809b830 con 0x7f25c4101af0 2024-04-24T23:43:29.406 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:29.406 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 msgr2=0x7f25ac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 0x7f25ac0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25b40099d0 tx=0x7f25b4008040 comp rx=0 tx=0).stop 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 msgr2=0x7f25c4199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f25b800b530 tx=0x7f25b800b840 comp rx=0 tx=0).stop 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 shutdown_connections 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f25ac0a2cb0 0x7f25ac0a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f25c41071a0 0x7f25c41a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25c4101130 0x7f25c41998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 --2- 172.21.15.107:0/3437355210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25c4101af0 0x7f25c4199e10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 >> 172.21.15.107:0/3437355210 conn(0x7f25c40780e0 msgr2=0x7f25c40ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 shutdown_connections 2024-04-24T23:43:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:29.408+0000 7f25c8b34700 1 -- 172.21.15.107:0/3437355210 wait complete. 2024-04-24T23:43:29.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:29 smithi195 ceph-mon[33019]: pgmap v309: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.8 KiB/s wr, 23 op/s 2024-04-24T23:43:29.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:29 smithi170 ceph-mon[32714]: pgmap v309: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.8 KiB/s wr, 23 op/s 2024-04-24T23:43:30.170 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-24T23:43:30.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T23:43:30.410 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:30.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:30 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3437355210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:30.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:30 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3437355210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:30.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:30 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3437355210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:31.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:31 smithi195 ceph-mon[33019]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.9 KiB/s wr, 23 op/s 2024-04-24T23:43:31.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:31 smithi170 ceph-mon[32714]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.9 KiB/s wr, 23 op/s 2024-04-24T23:43:31.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:31 smithi107 ceph-mon[29118]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.9 KiB/s wr, 23 op/s 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- 172.21.15.107:0/1721444613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a41024e0 msgr2=0x7f17a41028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/1721444613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a41024e0 0x7f17a41028c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1794009990 tx=0x7f1794009ca0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- 172.21.15.107:0/1721444613 shutdown_connections 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/1721444613 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a4107570 0x7f17a410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/1721444613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a41024e0 0x7f17a41028c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/1721444613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a4106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- 172.21.15.107:0/1721444613 >> 172.21.15.107:0/1721444613 conn(0x7f17a4079390 msgr2=0x7f17a40797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- 172.21.15.107:0/1721444613 shutdown_connections 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- 172.21.15.107:0/1721444613 wait complete. 2024-04-24T23:43:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 Processor -- start 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.204+0000 7f17a9ebe700 1 -- start start 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a41024e0 0x7f17a419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17a4112630 con 0x7f17a4107570 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41124b0 con 0x7f17a4102e90 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a9ebe700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41127b0 con 0x7f17a41024e0 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43314/0 (socket says 172.21.15.107:43314) 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 -- 172.21.15.107:0/207661901 learned_addr learned my addr 172.21.15.107:0/207661901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a37fe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a41024e0 0x7f17a419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f179affd700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 -- 172.21.15.107:0/207661901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a41024e0 msgr2=0x7f17a419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a41024e0 0x7f17a419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 -- 172.21.15.107:0/207661901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 msgr2=0x7f17a419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 -- 172.21.15.107:0/207661901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a41990d0 con 0x7f17a4107570 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f179affd700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.205+0000 7f17a3fff700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f179c00b970 tx=0x7f179c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.206+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c019070 con 0x7f17a4107570 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.206+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a41993c0 con 0x7f17a4107570 2024-04-24T23:43:32.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.206+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179c00ce10 con 0x7f17a4107570 2024-04-24T23:43:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.206+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c007370 con 0x7f17a4107570 2024-04-24T23:43:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.206+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a5680 con 0x7f17a4107570 2024-04-24T23:43:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.207+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a4068f20 con 0x7f17a4107570 2024-04-24T23:43:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.208+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f179c01c070 con 0x7f17a4107570 2024-04-24T23:43:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.208+0000 7f17a17fa700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.208+0000 7f17a37fe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.208+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f179c0ceb10 con 0x7f17a4107570 2024-04-24T23:43:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.209+0000 7f17a37fe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 0x7f17900a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f179400b5c0 tx=0x7f1794005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.212+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f179c09b7e0 con 0x7f17a4107570 2024-04-24T23:43:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.407+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17a4199ea0 con 0x7f17a4107570 2024-04-24T23:43:32.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.409+0000 7f17a17fa700 1 -- 172.21.15.107:0/207661901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f179c09b0e0 con 0x7f17a4107570 2024-04-24T23:43:32.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:43:32.409 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:42:55.655469+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 msgr2=0x7f17900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 0x7f17900a51f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f179400b5c0 tx=0x7f1794005960 comp rx=0 tx=0).stop 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 msgr2=0x7f17a4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f179c00b970 tx=0x7f179c00bc80 comp rx=0 tx=0).stop 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 shutdown_connections 2024-04-24T23:43:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f17900a2d30 0x7f17900a51f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17a4107570 0x7f17a4198970 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f17a41024e0 0x7f17a419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 --2- 172.21.15.107:0/207661901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17a4102e90 0x7f17a419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 >> 172.21.15.107:0/207661901 conn(0x7f17a4079390 msgr2=0x7f17a41086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 shutdown_connections 2024-04-24T23:43:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:32.411+0000 7f17a9ebe700 1 -- 172.21.15.107:0/207661901 wait complete. 2024-04-24T23:43:33.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-24T23:43:33.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-24T23:43:33.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-24T23:43:33.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-24T23:43:33.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-24T23:43:33.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-24T23:43:33.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-24T23:43:33.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-24T23:43:33.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-24T23:43:33.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-24T23:43:33.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-24T23:43:33.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-24T23:43:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:33 smithi195 ceph-mon[33019]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-24T23:43:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:33 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/207661901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:33 smithi170 ceph-mon[32714]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-24T23:43:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:33 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/207661901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:33.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:33 smithi107 ceph-mon[29118]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-24T23:43:33.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:33 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/207661901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:43:33.725 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:33.760 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:35 smithi195 ceph-mon[33019]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-24T23:43:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:35 smithi170 ceph-mon[32714]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-24T23:43:35.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:35 smithi107 ceph-mon[29118]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-24T23:43:36.157 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.158 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.165 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.490 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.491 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:36.974 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:37.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:37 smithi107 ceph-mon[29118]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-24T23:43:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:37 smithi195 ceph-mon[33019]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-24T23:43:37.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:37 smithi170 ceph-mon[32714]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 -- 172.21.15.107:0/2462038803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806b5b0 msgr2=0x7f8908106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 --2- 172.21.15.107:0/2462038803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806b5b0 0x7f8908106ee0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f88f80099c0 tx=0x7f88f8009cd0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 -- 172.21.15.107:0/2462038803 shutdown_connections 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 --2- 172.21.15.107:0/2462038803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 0x7f890810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 --2- 172.21.15.107:0/2462038803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890806ac00 0x7f890806afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 --2- 172.21.15.107:0/2462038803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806b5b0 0x7f8908106ee0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 -- 172.21.15.107:0/2462038803 >> 172.21.15.107:0/2462038803 conn(0x7f8908077180 msgr2=0x7f8908077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 -- 172.21.15.107:0/2462038803 shutdown_connections 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.940+0000 7f890fbdf700 1 -- 172.21.15.107:0/2462038803 wait complete. 2024-04-24T23:43:38.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 Processor -- start 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 -- start start 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 0x7f890819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890806b5b0 0x7f890819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 0x7f8908198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8908112580 con 0x7f890806ac00 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8908112400 con 0x7f8908107610 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890fbdf700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8908112700 con 0x7f890806b5b0 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890d97b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 0x7f890819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890d97b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 0x7f890819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44602/0 (socket says 172.21.15.107:44602) 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890d97b700 1 -- 172.21.15.107:0/1424660209 learned_addr learned my addr 172.21.15.107:0/1424660209 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890d17a700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890806b5b0 0x7f890819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.941+0000 7f890e17c700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 0x7f8908198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 -- 172.21.15.107:0/1424660209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890806b5b0 msgr2=0x7f890819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f890806b5b0 0x7f890819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 -- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 msgr2=0x7f8908198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 0x7f8908198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8908199000 con 0x7f890806ac00 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890e17c700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8908107610 0x7f8908198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890d97b700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 0x7f890819e650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f890400d980 tx=0x7f890400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904017070 con 0x7f890806ac00 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8904005020 con 0x7f890806ac00 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89081992f0 con 0x7f890806ac00 2024-04-24T23:43:38.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904009bc0 con 0x7f890806ac00 2024-04-24T23:43:38.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.942+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89081a54b0 con 0x7f890806ac00 2024-04-24T23:43:38.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.943+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8908062980 con 0x7f890806ac00 2024-04-24T23:43:38.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.944+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f89040051d0 con 0x7f890806ac00 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f88feffd700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88f40a2cb0 0x7f88f40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f890d17a700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88f40a2cb0 0x7f88f40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f89040cf8d0 con 0x7f890806ac00 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f88feffd700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] conn(0x7f88f40ad970 0x7f88f40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89040044e0 con 0x7f88f40ad970 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f89040cfd20 con 0x7f890806ac00 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.945+0000 7f890d17a700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88f40a2cb0 0x7f88f40a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009990 tx=0x7f88f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.946+0000 7f890e17c700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] conn(0x7f88f40ad970 0x7f88f40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:38.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.946+0000 7f890e17c700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] conn(0x7f88f40ad970 0x7f88f40afdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:38.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.947+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== osd.4 v2:172.21.15.170:6808/2310043947 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89040044e0 con 0x7f88f40ad970 2024-04-24T23:43:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.964+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 --> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8908062e00 con 0x7f88f40ad970 2024-04-24T23:43:38.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.964+0000 7f88feffd700 1 -- 172.21.15.107:0/1424660209 <== osd.4 v2:172.21.15.170:6808/2310043947 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8908062e00 con 0x7f88f40ad970 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] conn(0x7f88f40ad970 msgr2=0x7f88f40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] conn(0x7f88f40ad970 0x7f88f40afdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88f40a2cb0 msgr2=0x7f88f40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88f40a2cb0 0x7f88f40a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009990 tx=0x7f88f8005fb0 comp rx=0 tx=0).stop 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 msgr2=0x7f890819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 --2- 172.21.15.107:0/1424660209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f890806ac00 0x7f890819e650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f890400d980 tx=0x7f890400dc90 comp rx=0 tx=0).stop 2024-04-24T23:43:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890e17c700 1 -- 172.21.15.107:0/1424660209 reap_dead start 2024-04-24T23:43:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 shutdown_connections 2024-04-24T23:43:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 >> 172.21.15.107:0/1424660209 conn(0x7f8908077180 msgr2=0x7f89080ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 shutdown_connections 2024-04-24T23:43:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:38.965+0000 7f890fbdf700 1 -- 172.21.15.107:0/1424660209 wait complete. 2024-04-24T23:43:39.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:39 smithi107 ceph-mon[29118]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-24T23:43:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 -- 172.21.15.107:0/1243702414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102550 msgr2=0x7f7be4102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 --2- 172.21.15.107:0/1243702414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102550 0x7f7be4102930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400b0d0 tx=0x7f7bd400b3e0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 -- 172.21.15.107:0/1243702414 shutdown_connections 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 --2- 172.21.15.107:0/1243702414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7be4107540 0x7f7be410f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 --2- 172.21.15.107:0/1243702414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102550 0x7f7be4102930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 --2- 172.21.15.107:0/1243702414 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7be4102f00 0x7f7be4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.226+0000 7f7bec223700 1 -- 172.21.15.107:0/1243702414 >> 172.21.15.107:0/1243702414 conn(0x7f7be40793f0 msgr2=0x7f7be4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- 172.21.15.107:0/1243702414 shutdown_connections 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- 172.21.15.107:0/1243702414 wait complete. 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 Processor -- start 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- start start 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7be4102550 0x7f7be419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 0x7f7be419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7be4107540 0x7f7be41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7be41125f0 con 0x7f7be4102f00 2024-04-24T23:43:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4112470 con 0x7f7be4102550 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7bec223700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4112770 con 0x7f7be4107540 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.227+0000 7f7be97be700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 0x7f7be419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 0x7f7be419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44614/0 (socket says 172.21.15.107:44614) 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 -- 172.21.15.107:0/3723736564 learned_addr learned my addr 172.21.15.107:0/3723736564 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be9fbf700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7be4102550 0x7f7be419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 -- 172.21.15.107:0/3723736564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7be4107540 msgr2=0x7f7be41988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7bea7c0700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7be4107540 0x7f7be41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7be4107540 0x7f7be41988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 -- 172.21.15.107:0/3723736564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7be4102550 msgr2=0x7f7be419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7be4102550 0x7f7be419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be4199030 con 0x7f7be4102f00 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be97be700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 0x7f7be419ec50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800d980 tx=0x7f7bd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8017070 con 0x7f7be4102f00 2024-04-24T23:43:39.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be4199320 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd8009e10 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800f620 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.228+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be41a55e0 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.229+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7be4062940 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.230+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f7bd8010460 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be2ffd700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7bcc0a2cb0 0x7f7bcc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be9fbf700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7bcc0a2cb0 0x7f7bcc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7bd80cf340 con 0x7f7be4102f00 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be2ffd700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] conn(0x7f7bcc0ad970 0x7f7bcc0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:39.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bcc0b0480 con 0x7f7bcc0ad970 2024-04-24T23:43:39.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7bd80cf790 con 0x7f7be4102f00 2024-04-24T23:43:39.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7bea7c0700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] conn(0x7f7bcc0ad970 0x7f7bcc0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:39.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7be9fbf700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7bcc0a2cb0 0x7f7bcc0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009fd0 tx=0x7f7bd400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:39.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.231+0000 7f7bea7c0700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] conn(0x7f7bcc0ad970 0x7f7bcc0afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:39.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.232+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== osd.9 v2:172.21.15.107:6826/150997445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7bcc0b0480 con 0x7f7bcc0ad970 2024-04-24T23:43:39.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.252+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 --> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7be4062d40 con 0x7f7bcc0ad970 2024-04-24T23:43:39.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.252+0000 7f7be2ffd700 1 -- 172.21.15.107:0/3723736564 <== osd.9 v2:172.21.15.107:6826/150997445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7be4062d40 con 0x7f7bcc0ad970 2024-04-24T23:43:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] conn(0x7f7bcc0ad970 msgr2=0x7f7bcc0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] conn(0x7f7bcc0ad970 0x7f7bcc0afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7bcc0a2cb0 msgr2=0x7f7bcc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7bcc0a2cb0 0x7f7bcc0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009fd0 tx=0x7f7bd400b040 comp rx=0 tx=0).stop 2024-04-24T23:43:39.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 msgr2=0x7f7be419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 --2- 172.21.15.107:0/3723736564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7be4102f00 0x7f7be419ec50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800d980 tx=0x7f7bd800dc90 comp rx=0 tx=0).stop 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bea7c0700 1 -- 172.21.15.107:0/3723736564 reap_dead start 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 shutdown_connections 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 >> 172.21.15.107:0/3723736564 conn(0x7f7be40793f0 msgr2=0x7f7be41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 shutdown_connections 2024-04-24T23:43:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:39.253+0000 7f7bec223700 1 -- 172.21.15.107:0/3723736564 wait complete. 2024-04-24T23:43:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:39 smithi195 ceph-mon[33019]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-24T23:43:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:39 smithi170 ceph-mon[32714]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-24T23:43:41.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:41 smithi107 ceph-mon[29118]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:41 smithi195 ceph-mon[33019]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:41 smithi170 ceph-mon[32714]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.930+0000 7f64723ae700 1 -- 172.21.15.107:0/1187033601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c102ca0 msgr2=0x7f646c10f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.930+0000 7f64723ae700 1 --2- 172.21.15.107:0/1187033601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c102ca0 0x7f646c10f110 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f646400b0d0 tx=0x7f646400b3e0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 -- 172.21.15.107:0/1187033601 shutdown_connections 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 --2- 172.21.15.107:0/1187033601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c102ca0 0x7f646c10f110 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 --2- 172.21.15.107:0/1187033601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f646c1082f0 0x7f646c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 --2- 172.21.15.107:0/1187033601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646c1022e0 0x7f646c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 -- 172.21.15.107:0/1187033601 >> 172.21.15.107:0/1187033601 conn(0x7f646c079390 msgr2=0x7f646c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 -- 172.21.15.107:0/1187033601 shutdown_connections 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 -- 172.21.15.107:0/1187033601 wait complete. 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 Processor -- start 2024-04-24T23:43:42.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.931+0000 7f64723ae700 1 -- start start 2024-04-24T23:43:42.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646c1022e0 0x7f646c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:42.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f646c102ca0 0x7f646c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:42.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 0x7f646c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:42.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f646c112570 con 0x7f646c1082f0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f646c1123f0 con 0x7f646c1022e0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f646c1126f0 con 0x7f646c102ca0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 0x7f646c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 0x7f646c074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44622/0 (socket says 172.21.15.107:44622) 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 -- 172.21.15.107:0/2359344134 learned_addr learned my addr 172.21.15.107:0/2359344134 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f646bfff700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646c1022e0 0x7f646c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f646b7fe700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f646c102ca0 0x7f646c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 -- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f646c102ca0 msgr2=0x7f646c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f646c102ca0 0x7f646c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 -- 172.21.15.107:0/2359344134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646c1022e0 msgr2=0x7f646c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646c1022e0 0x7f646c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c1a6d80 con 0x7f646c1082f0 2024-04-24T23:43:42.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f647094b700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 0x7f646c074fb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6464006070 tx=0x7f6464004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464021070 con 0x7f646c1082f0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6464007b80 con 0x7f646c1082f0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.932+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646c1a7010 con 0x7f646c1082f0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.933+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64640052a0 con 0x7f646c1082f0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.933+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646c1a74c0 con 0x7f646c1082f0 2024-04-24T23:43:42.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.933+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f646c1016e0 con 0x7f646c1082f0 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.934+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6464024070 con 0x7f646c1082f0 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f64697fa700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64540a2cb0 0x7f64540a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f646bfff700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64540a2cb0 0x7f64540a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f646400f020 con 0x7f646c1082f0 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f64697fa700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] conn(0x7f64540ad970 0x7f64540afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64540b0480 con 0x7f64540ad970 2024-04-24T23:43:42.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.935+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f64640d7050 con 0x7f646c1082f0 2024-04-24T23:43:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.936+0000 7f646bfff700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64540a2cb0 0x7f64540a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f645c00abd0 tx=0x7f645c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.936+0000 7f646b7fe700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] conn(0x7f64540ad970 0x7f64540afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.936+0000 7f646b7fe700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] conn(0x7f64540ad970 0x7f64540afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.937+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== osd.8 v2:172.21.15.195:6816/2912271513 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64540b0480 con 0x7f64540ad970 2024-04-24T23:43:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.954+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 --> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f646c1039a0 con 0x7f64540ad970 2024-04-24T23:43:42.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.954+0000 7f64697fa700 1 -- 172.21.15.107:0/2359344134 <== osd.8 v2:172.21.15.195:6816/2912271513 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f646c1039a0 con 0x7f64540ad970 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] conn(0x7f64540ad970 msgr2=0x7f64540afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] conn(0x7f64540ad970 0x7f64540afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64540a2cb0 msgr2=0x7f64540a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f64540a2cb0 0x7f64540a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f645c00abd0 tx=0x7f645c00a380 comp rx=0 tx=0).stop 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 msgr2=0x7f646c074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 --2- 172.21.15.107:0/2359344134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f646c1082f0 0x7f646c074fb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6464006070 tx=0x7f6464004d00 comp rx=0 tx=0).stop 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f647094b700 1 -- 172.21.15.107:0/2359344134 reap_dead start 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 shutdown_connections 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 >> 172.21.15.107:0/2359344134 conn(0x7f646c079390 msgr2=0x7f646c100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 shutdown_connections 2024-04-24T23:43:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:42.955+0000 7f64723ae700 1 -- 172.21.15.107:0/2359344134 wait complete. 2024-04-24T23:43:43.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:43 smithi107 ceph-mon[29118]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:43 smithi195 ceph-mon[33019]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:43.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:43 smithi170 ceph-mon[32714]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-24T23:43:45.143 INFO:teuthology.orchestra.run.smithi107.stdout:141733920827 2024-04-24T23:43:45.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-24T23:43:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- 172.21.15.107:0/3803516026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378107540 msgr2=0x7fe37810f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/3803516026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378107540 0x7fe37810f110 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe37000b0d0 tx=0x7fe37000b3e0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- 172.21.15.107:0/3803516026 shutdown_connections 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/3803516026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378107540 0x7fe37810f110 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/3803516026 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378102550 0x7fe378102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/3803516026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 0x7fe378106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- 172.21.15.107:0/3803516026 >> 172.21.15.107:0/3803516026 conn(0x7fe3780793f0 msgr2=0x7fe378079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- 172.21.15.107:0/3803516026 shutdown_connections 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- 172.21.15.107:0/3803516026 wait complete. 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 Processor -- start 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.198+0000 7fe37dfe7700 1 -- start start 2024-04-24T23:43:45.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 0x7fe37819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 0x7fe37819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378107540 0x7fe3781987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe378112530 con 0x7fe378102550 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe3781123b0 con 0x7fe378102f00 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe3781126b0 con 0x7fe378107540 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 0x7fe37819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 0x7fe37819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44646/0 (socket says 172.21.15.107:44646) 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 -- 172.21.15.107:0/2646641554 learned_addr learned my addr 172.21.15.107:0/2646641554 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe36effd700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 0x7fe37819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 -- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378107540 msgr2=0x7fe3781987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe377fff700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378107540 0x7fe3781987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378107540 0x7fe3781987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 -- 172.21.15.107:0/2646641554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 msgr2=0x7fe37819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 0x7fe37819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe378198fd0 con 0x7fe378102550 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe36effd700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe378102f00 0x7fe37819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe377fff700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe378107540 0x7fe3781987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:43:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3777fe700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 0x7fe37819e670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe36800d950 tx=0x7fe36800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe368016070 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3781992c0 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe368009b40 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36800f620 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.199+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3781a54d0 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.200+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe3780629c0 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe368019070 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe3640a2cb0 0x7fe3640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe36effd700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe3640a2cb0 0x7fe3640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe3680cf510 con 0x7fe378102550 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] conn(0x7fe3640ad970 0x7fe3640afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe368014e80 con 0x7fe3640ad970 2024-04-24T23:43:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.202+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe3680cf960 con 0x7fe378102550 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.203+0000 7fe36effd700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe3640a2cb0 0x7fe3640a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe360005fd0 tx=0x7fe360005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.203+0000 7fe377fff700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] conn(0x7fe3640ad970 0x7fe3640afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.203+0000 7fe377fff700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] conn(0x7fe3640ad970 0x7fe3640afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.204+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== osd.2 v2:172.21.15.195:6800/3967620404 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe368014e80 con 0x7fe3640ad970 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.220+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 --> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe378062e40 con 0x7fe3640ad970 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.221+0000 7fe3757fa700 1 -- 172.21.15.107:0/2646641554 <== osd.2 v2:172.21.15.195:6800/3967620404 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe378062e40 con 0x7fe3640ad970 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] conn(0x7fe3640ad970 msgr2=0x7fe3640afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] conn(0x7fe3640ad970 0x7fe3640afdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe3640a2cb0 msgr2=0x7fe3640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe3640a2cb0 0x7fe3640a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe360005fd0 tx=0x7fe360005d40 comp rx=0 tx=0).stop 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 msgr2=0x7fe37819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 --2- 172.21.15.107:0/2646641554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe378102550 0x7fe37819e670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe36800d950 tx=0x7fe36800dd10 comp rx=0 tx=0).stop 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe377fff700 1 -- 172.21.15.107:0/2646641554 reap_dead start 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 shutdown_connections 2024-04-24T23:43:45.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 >> 172.21.15.107:0/2646641554 conn(0x7fe3780793f0 msgr2=0x7fe378100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 shutdown_connections 2024-04-24T23:43:45.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:44.222+0000 7fe37dfe7700 1 -- 172.21.15.107:0/2646641554 wait complete. 2024-04-24T23:43:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:45 smithi107 ceph-mon[29118]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-24T23:43:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:45 smithi195 ceph-mon[33019]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-24T23:43:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:45 smithi170 ceph-mon[32714]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-24T23:43:46.380 INFO:teuthology.orchestra.run.smithi107.stdout:253403070503 2024-04-24T23:43:46.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-24T23:43:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:47 smithi195 ceph-mon[33019]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:43:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:47 smithi170 ceph-mon[32714]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:43:47.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:47 smithi107 ceph-mon[29118]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:43:48.078 INFO:teuthology.orchestra.run.smithi107.stdout:77309411398 2024-04-24T23:43:48.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-24T23:43:48.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-24T23:43:48.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:48 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-24T23:43:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-24T23:43:48.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-24T23:43:48.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:48 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-24T23:43:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T23:43:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-24T23:43:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-04-24T23:43:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-24T23:43:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-24T23:43:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-24T23:43:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-24T23:43:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-24T23:43:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:48 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-24T23:43:48.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 -- 172.21.15.107:0/1547504456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 msgr2=0x7fc92010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 --2- 172.21.15.107:0/1547504456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92010dfe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc91800b0a0 tx=0x7fc91800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 -- 172.21.15.107:0/1547504456 shutdown_connections 2024-04-24T23:43:48.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 --2- 172.21.15.107:0/1547504456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92010dfe0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 --2- 172.21.15.107:0/1547504456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9201071e0 0x7fc9201075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 --2- 172.21.15.107:0/1547504456 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc920101160 0x7fc9201015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.718+0000 7fc927767700 1 -- 172.21.15.107:0/1547504456 >> 172.21.15.107:0/1547504456 conn(0x7fc9200771a0 msgr2=0x7fc9200775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 -- 172.21.15.107:0/1547504456 shutdown_connections 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 -- 172.21.15.107:0/1547504456 wait complete. 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 Processor -- start 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 -- start start 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc920101160 0x7fc92019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.719+0000 7fc927767700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9201071e0 0x7fc9201976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9201114a0 con 0x7fc920101b20 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc920111320 con 0x7fc9201071e0 2024-04-24T23:43:48.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc920111620 con 0x7fc920101160 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92019db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38512/0 (socket says 172.21.15.107:38512) 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 -- 172.21.15.107:0/3784877821 learned_addr learned my addr 172.21.15.107:0/3784877821 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc925d04700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9201071e0 0x7fc9201976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc925503700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc920101160 0x7fc92019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 -- 172.21.15.107:0/3784877821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc920101160 msgr2=0x7fc92019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc920101160 0x7fc92019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 -- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9201071e0 msgr2=0x7fc9201976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9201071e0 0x7fc9201976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc920197fc0 con 0x7fc920101b20 2024-04-24T23:43:48.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc924d02700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92019db50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00b530 tx=0x7fc91c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c015020 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91c011d10 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9201982b0 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00f2e0 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.720+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9201a4590 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.721+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc920062980 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.722+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc91c018020 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc9167fc700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc90c0a2cb0 0x7fc90c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc925503700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc90c0a2cb0 0x7fc90c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fc91c0cf2f0 con 0x7fc920101b20 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc9167fc700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] conn(0x7fc90c0ade10 0x7fc90c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc91c013ec0 con 0x7fc90c0ade10 2024-04-24T23:43:48.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc91c0cf740 con 0x7fc920101b20 2024-04-24T23:43:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.723+0000 7fc925503700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc90c0a2cb0 0x7fc90c0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc910005950 tx=0x7fc9100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.724+0000 7fc925d04700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] conn(0x7fc90c0ade10 0x7fc90c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.724+0000 7fc925d04700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] conn(0x7fc90c0ade10 0x7fc90c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:48.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.725+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== osd.1 v2:172.21.15.170:6800/80173128 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc91c013ec0 con 0x7fc90c0ade10 2024-04-24T23:43:48.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.742+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 --> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc920062d80 con 0x7fc90c0ade10 2024-04-24T23:43:48.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.742+0000 7fc9167fc700 1 -- 172.21.15.107:0/3784877821 <== osd.1 v2:172.21.15.170:6800/80173128 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc920062d80 con 0x7fc90c0ade10 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] conn(0x7fc90c0ade10 msgr2=0x7fc90c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] conn(0x7fc90c0ade10 0x7fc90c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc90c0a2cb0 msgr2=0x7fc90c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc90c0a2cb0 0x7fc90c0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc910005950 tx=0x7fc9100058e0 comp rx=0 tx=0).stop 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 msgr2=0x7fc92019db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 --2- 172.21.15.107:0/3784877821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc920101b20 0x7fc92019db50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00b530 tx=0x7fc91c00b840 comp rx=0 tx=0).stop 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc925d04700 1 -- 172.21.15.107:0/3784877821 reap_dead start 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 shutdown_connections 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 >> 172.21.15.107:0/3784877821 conn(0x7fc9200771a0 msgr2=0x7fc9200ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:48.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 shutdown_connections 2024-04-24T23:43:48.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:48.743+0000 7fc927767700 1 -- 172.21.15.107:0/3784877821 wait complete. 2024-04-24T23:43:49.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- 172.21.15.107:0/80231623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 msgr2=0x7f07a41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/80231623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a41015b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0794009990 tx=0x7f0794009ca0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- 172.21.15.107:0/80231623 shutdown_connections 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/80231623 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07a4101af0 0x7f07a410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/80231623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a41071a0 0x7f07a4107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/80231623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a41015b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- 172.21.15.107:0/80231623 >> 172.21.15.107:0/80231623 conn(0x7f07a40780e0 msgr2=0x7f07a40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- 172.21.15.107:0/80231623 shutdown_connections 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- 172.21.15.107:0/80231623 wait complete. 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 Processor -- start 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.210+0000 7f07ac0c2700 1 -- start start 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a4101af0 0x7f07a4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07a41071a0 0x7f07a41a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:49.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4111440 con 0x7f07a4101130 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41112c0 con 0x7f07a4101af0 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07ac0c2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41115c0 con 0x7f07a41071a0 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a4199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38536/0 (socket says 172.21.15.107:38536) 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 -- 172.21.15.107:0/1174024025 learned_addr learned my addr 172.21.15.107:0/1174024025 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a965d700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a4101af0 0x7f07a4199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07aa65f700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07a41071a0 0x7f07a41a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 -- 172.21.15.107:0/1174024025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07a41071a0 msgr2=0x7f07a41a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07a41071a0 0x7f07a41a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 -- 172.21.15.107:0/1174024025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a4101af0 msgr2=0x7f07a4199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a4101af0 0x7f07a4199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a41a4dc0 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.211+0000 7f07a9e5e700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a4199800 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f07a000d980 tx=0x7f07a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.212+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0016070 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.212+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a41a4ff0 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.212+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a41a5440 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.212+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07a0004d10 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.212+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0005670 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.213+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07a40628c0 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.215+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f07a0004750 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.215+0000 7f079affd700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07900a2d30 0x7f07900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f07a965d700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07900a2d30 0x7f07900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f07a00cf7c0 con 0x7f07a4101130 2024-04-24T23:43:49.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f079affd700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] conn(0x7f07900ade10 0x7f07900b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07a0015ec0 con 0x7f07900ade10 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f07aa65f700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] conn(0x7f07900ade10 0x7f07900b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.216+0000 7f07a965d700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07900a2d30 0x7f07900a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f079400b2e0 tx=0x7f0794005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.217+0000 7f07aa65f700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] conn(0x7f07900ade10 0x7f07900b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.221+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f07a0097e00 con 0x7f07a4101130 2024-04-24T23:43:49.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.221+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== osd.6 v2:172.21.15.107:6818/3398193587 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07a0015ec0 con 0x7f07900ade10 2024-04-24T23:43:49.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 --> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f07a4062d70 con 0x7f07900ade10 2024-04-24T23:43:49.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f079affd700 1 -- 172.21.15.107:0/1174024025 <== osd.6 v2:172.21.15.107:6818/3398193587 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f07a4062d70 con 0x7f07900ade10 2024-04-24T23:43:49.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] conn(0x7f07900ade10 msgr2=0x7f07900b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] conn(0x7f07900ade10 0x7f07900b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:49.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07900a2d30 msgr2=0x7f07900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07900a2d30 0x7f07900a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f079400b2e0 tx=0x7f0794005960 comp rx=0 tx=0).stop 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 msgr2=0x7f07a4199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 --2- 172.21.15.107:0/1174024025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07a4101130 0x7f07a4199800 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f07a000d980 tx=0x7f07a000dc90 comp rx=0 tx=0).stop 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07aa65f700 1 -- 172.21.15.107:0/1174024025 reap_dead start 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 shutdown_connections 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.243+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 >> 172.21.15.107:0/1174024025 conn(0x7f07a40780e0 msgr2=0x7f07a40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.244+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 shutdown_connections 2024-04-24T23:43:49.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:49.244+0000 7f07ac0c2700 1 -- 172.21.15.107:0/1174024025 wait complete. 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-24T23:43:49.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-24T23:43:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-24T23:43:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-24T23:43:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T23:43:49.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-24T23:43:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-24T23:43:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-24T23:43:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-24T23:43:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T23:43:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:43:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-24T23:43:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-24T23:43:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T23:43:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-24T23:43:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T23:43:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:43:50.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 -- 172.21.15.107:0/1532829496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c06a8e0 msgr2=0x7f275c10f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 --2- 172.21.15.107:0/1532829496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c06a8e0 0x7f275c10f0f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f275400b0a0 tx=0x7f275400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 -- 172.21.15.107:0/1532829496 shutdown_connections 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 --2- 172.21.15.107:0/1532829496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c06a8e0 0x7f275c10f0f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 --2- 172.21.15.107:0/1532829496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c1082b0 0x7f275c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 --2- 172.21.15.107:0/1532829496 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f275c069f40 0x7f275c06a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 -- 172.21.15.107:0/1532829496 >> 172.21.15.107:0/1532829496 conn(0x7f275c0780e0 msgr2=0x7f275c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 -- 172.21.15.107:0/1532829496 shutdown_connections 2024-04-24T23:43:50.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.252+0000 7f27621d5700 1 -- 172.21.15.107:0/1532829496 wait complete. 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.253+0000 7f27621d5700 1 Processor -- start 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.253+0000 7f27621d5700 1 -- start start 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 0x7f275c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c06a8e0 0x7f275c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f275c1082b0 0x7f275c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f275c112500 con 0x7f275c069f40 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f275c112380 con 0x7f275c06a8e0 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f27621d5700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f275c112680 con 0x7f275c1082b0 2024-04-24T23:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 0x7f275c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 0x7f275c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38542/0 (socket says 172.21.15.107:38542) 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 -- 172.21.15.107:0/4092874363 learned_addr learned my addr 172.21.15.107:0/4092874363 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275affd700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c06a8e0 0x7f275c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275bfff700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f275c1082b0 0x7f275c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 -- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f275c1082b0 msgr2=0x7f275c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f275c1082b0 0x7f275c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 -- 172.21.15.107:0/4092874363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c06a8e0 msgr2=0x7f275c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c06a8e0 0x7f275c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c199020 con 0x7f275c069f40 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275affd700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f275c06a8e0 0x7f275c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f275b7fe700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 0x7f275c19e620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f275000d950 tx=0x7f275000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.254+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750016070 con 0x7f275c069f40 2024-04-24T23:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.255+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275c199310 con 0x7f275c069f40 2024-04-24T23:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.255+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2750009b40 con 0x7f275c069f40 2024-04-24T23:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.255+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275000f620 con 0x7f275c069f40 2024-04-24T23:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.255+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275c1a55a0 con 0x7f275c069f40 2024-04-24T23:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.255+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f275c062940 con 0x7f275c069f40 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2750019070 con 0x7f275c069f40 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f27440a2cb0 0x7f27440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f275affd700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f27440a2cb0 0x7f27440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f27500cf5e0 con 0x7f275c069f40 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] conn(0x7f27440ade10 0x7f27440b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2750014e80 con 0x7f27440ade10 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.257+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f27500cfa30 con 0x7f275c069f40 2024-04-24T23:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.258+0000 7f275bfff700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] conn(0x7f27440ade10 0x7f27440b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.258+0000 7f275affd700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f27440a2cb0 0x7f27440a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f274c005fd0 tx=0x7f274c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.258+0000 7f275bfff700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] conn(0x7f27440ade10 0x7f27440b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.259+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== osd.5 v2:172.21.15.195:6808/1022606297 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2750014e80 con 0x7f27440ade10 2024-04-24T23:43:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.275+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 --> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f275c062e00 con 0x7f27440ade10 2024-04-24T23:43:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.275+0000 7f2758ff9700 1 -- 172.21.15.107:0/4092874363 <== osd.5 v2:172.21.15.195:6808/1022606297 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f275c062e00 con 0x7f27440ade10 2024-04-24T23:43:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] conn(0x7f27440ade10 msgr2=0x7f27440b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] conn(0x7f27440ade10 0x7f27440b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f27440a2cb0 msgr2=0x7f27440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f27440a2cb0 0x7f27440a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f274c005fd0 tx=0x7f274c005e30 comp rx=0 tx=0).stop 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 msgr2=0x7f275c19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 --2- 172.21.15.107:0/4092874363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f275c069f40 0x7f275c19e620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f275000d950 tx=0x7f275000dd10 comp rx=0 tx=0).stop 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f275bfff700 1 -- 172.21.15.107:0/4092874363 reap_dead start 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 shutdown_connections 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 >> 172.21.15.107:0/4092874363 conn(0x7f275c0780e0 msgr2=0x7f275c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 shutdown_connections 2024-04-24T23:43:50.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:50.276+0000 7f27621d5700 1 -- 172.21.15.107:0/4092874363 wait complete. 2024-04-24T23:43:50.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:50 smithi107 ceph-mon[29118]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T23:43:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:50 smithi195 ceph-mon[33019]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T23:43:50.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:50 smithi170 ceph-mon[32714]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T23:43:51.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:51 smithi107 ceph-mon[29118]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-24T23:43:51.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:51 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.365 INFO:teuthology.orchestra.run.smithi107.stdout:163208757304 2024-04-24T23:43:51.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-24T23:43:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:51 smithi195 ceph-mon[33019]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-24T23:43:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:51 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.582 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:51.582 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:51 smithi170 ceph-mon[32714]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-24T23:43:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:51 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:43:52.020 INFO:teuthology.orchestra.run.smithi107.stdout:55834574923 2024-04-24T23:43:52.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-24T23:43:52.492 INFO:teuthology.orchestra.run.smithi107.stdout:287762808866 2024-04-24T23:43:52.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-24T23:43:52.728 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:52.736 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:53.201 INFO:teuthology.orchestra.run.smithi107.stdout:197568495668 2024-04-24T23:43:53.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-24T23:43:53.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:43:53.487 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:53 smithi107 ceph-mon[29118]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 12 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:43:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:53 smithi195 ceph-mon[33019]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 12 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:43:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:53 smithi170 ceph-mon[32714]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 383 B/s wr, 12 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 -- 172.21.15.107:0/3377175119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 msgr2=0x7f86e406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/3377175119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e406ac30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009990 tx=0x7f86cc009ca0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 -- 172.21.15.107:0/3377175119 shutdown_connections 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/3377175119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 0x7f86e410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/3377175119 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86e41082a0 0x7f86e4108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/3377175119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e406ac30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 -- 172.21.15.107:0/3377175119 >> 172.21.15.107:0/3377175119 conn(0x7f86e40780a0 msgr2=0x7f86e40784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:55.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 -- 172.21.15.107:0/3377175119 shutdown_connections 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.320+0000 7f86ea0e6700 1 -- 172.21.15.107:0/3377175119 wait complete. 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 Processor -- start 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 -- start start 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 0x7f86e419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86e41082a0 0x7f86e4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4112500 con 0x7f86e406a7d0 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4112380 con 0x7f86e406b170 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86ea0e6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4112680 con 0x7f86e41082a0 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86e37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86e37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38552/0 (socket says 172.21.15.107:38552) 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86e37fe700 1 -- 172.21.15.107:0/1265697095 learned_addr learned my addr 172.21.15.107:0/1265697095 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:55.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86daffd700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 0x7f86e419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.321+0000 7f86e3fff700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86e41082a0 0x7f86e4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 -- 172.21.15.107:0/1265697095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86e41082a0 msgr2=0x7f86e4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f86e41082a0 0x7f86e4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 -- 172.21.15.107:0/1265697095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 msgr2=0x7f86e419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 0x7f86e419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e4199040 con 0x7f86e406a7d0 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86daffd700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406b170 0x7f86e419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e37fe700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e419e690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f86d400d980 tx=0x7f86d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:55.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4017070 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86d4005020 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e4199330 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d400fbf0 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.322+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e41a54f0 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.323+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86e4062980 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.324+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f86d40051d0 con 0x7f86e406a7d0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e17fa700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86d00a2cb0 0x7f86d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86daffd700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86d00a2cb0 0x7f86d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:55.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f86d40cf830 con 0x7f86e406a7d0 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e17fa700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] conn(0x7f86d00ade10 0x7f86d00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86d40044e0 con 0x7f86d00ade10 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f86d40cfc80 con 0x7f86e406a7d0 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86e3fff700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] conn(0x7f86d00ade10 0x7f86d00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.325+0000 7f86daffd700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86d00a2cb0 0x7f86d00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00b2e0 tx=0x7f86cc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.326+0000 7f86e3fff700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] conn(0x7f86d00ade10 0x7f86d00b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:55.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.326+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== osd.3 v2:172.21.15.107:6810/3900016553 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86d40044e0 con 0x7f86d00ade10 2024-04-24T23:43:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.342+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 --> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f86e4062d80 con 0x7f86d00ade10 2024-04-24T23:43:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86e17fa700 1 -- 172.21.15.107:0/1265697095 <== osd.3 v2:172.21.15.107:6810/3900016553 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f86e4062d80 con 0x7f86d00ade10 2024-04-24T23:43:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] conn(0x7f86d00ade10 msgr2=0x7f86d00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] conn(0x7f86d00ade10 0x7f86d00b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86d00a2cb0 msgr2=0x7f86d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f86d00a2cb0 0x7f86d00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00b2e0 tx=0x7f86cc005960 comp rx=0 tx=0).stop 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 msgr2=0x7f86e419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 --2- 172.21.15.107:0/1265697095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86e406a7d0 0x7f86e419e690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f86d400d980 tx=0x7f86d400dc90 comp rx=0 tx=0).stop 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86e3fff700 1 -- 172.21.15.107:0/1265697095 reap_dead start 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 shutdown_connections 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 >> 172.21.15.107:0/1265697095 conn(0x7f86e40780a0 msgr2=0x7f86e40ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 shutdown_connections 2024-04-24T23:43:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:55.343+0000 7f86ea0e6700 1 -- 172.21.15.107:0/1265697095 wait complete. 2024-04-24T23:43:55.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:55 smithi107 ceph-mon[29118]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-24T23:43:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:55 smithi195 ceph-mon[33019]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-24T23:43:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:55 smithi170 ceph-mon[32714]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-24T23:43:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 -- 172.21.15.107:0/2465540347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b006a8e0 msgr2=0x7f07b010f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 --2- 172.21.15.107:0/2465540347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b006a8e0 0x7f07b010f0f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b0a0 tx=0x7f07a800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 -- 172.21.15.107:0/2465540347 shutdown_connections 2024-04-24T23:43:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 --2- 172.21.15.107:0/2465540347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b006a8e0 0x7f07b010f0f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 --2- 172.21.15.107:0/2465540347 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07b01082b0 0x7f07b0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 --2- 172.21.15.107:0/2465540347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07b0069f40 0x7f07b006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 -- 172.21.15.107:0/2465540347 >> 172.21.15.107:0/2465540347 conn(0x7f07b00780e0 msgr2=0x7f07b00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 -- 172.21.15.107:0/2465540347 shutdown_connections 2024-04-24T23:43:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.098+0000 7f07b55e1700 1 -- 172.21.15.107:0/2465540347 wait complete. 2024-04-24T23:43:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 Processor -- start 2024-04-24T23:43:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 -- start start 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07b0069f40 0x7f07b019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07b006a8e0 0x7f07b019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 0x7f07b0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0112550 con 0x7f07b01082b0 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f07b01123d0 con 0x7f07b006a8e0 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07b55e1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f07b01126d0 con 0x7f07b0069f40 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07af7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 0x7f07b0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07af7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 0x7f07b0198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58642/0 (socket says 172.21.15.107:58642) 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.099+0000 7f07af7fe700 1 -- 172.21.15.107:0/3023490601 learned_addr learned my addr 172.21.15.107:0/3023490601 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07aeffd700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07b0069f40 0x7f07b019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07ae7fc700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07b006a8e0 0x7f07b019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 -- 172.21.15.107:0/3023490601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07b0069f40 msgr2=0x7f07b019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07b0069f40 0x7f07b019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 -- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07b006a8e0 msgr2=0x7f07b019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07b006a8e0 0x7f07b019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b0199010 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07af7fe700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 0x7f07b0198730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d010 tx=0x7f07a800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8007b60 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b01992a0 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b01a5590 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07a8007d00 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.100+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8004650 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.101+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f079c001030 con 0x7f07b01082b0 2024-04-24T23:43:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.103+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f07a8024070 con 0x7f07b01082b0 2024-04-24T23:43:57.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.103+0000 7f0797fff700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07980a2df0 0x7f07980a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f07aeffd700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07980a2df0 0x7f07980a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f07a800f030 con 0x7f07b01082b0 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f0797fff700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] conn(0x7f07980adf50 0x7f07980b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07a800fec0 con 0x7f07980adf50 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f07a80d07b0 con 0x7f07b01082b0 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f07aeffd700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07980a2df0 0x7f07980a52b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005fd0 tx=0x7f07a000af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.104+0000 7f07ae7fc700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] conn(0x7f07980adf50 0x7f07980b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.105+0000 7f07ae7fc700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] conn(0x7f07980adf50 0x7f07980b03f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.106+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== osd.7 v2:172.21.15.170:6816/2352399844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07a800fec0 con 0x7f07980adf50 2024-04-24T23:43:57.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.123+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 --> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f079c002e60 con 0x7f07980adf50 2024-04-24T23:43:57.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.123+0000 7f0797fff700 1 -- 172.21.15.107:0/3023490601 <== osd.7 v2:172.21.15.170:6816/2352399844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f079c002e60 con 0x7f07980adf50 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.123+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] conn(0x7f07980adf50 msgr2=0x7f07980b03f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.123+0000 7f07b55e1700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] conn(0x7f07980adf50 0x7f07980b03f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07980a2df0 msgr2=0x7f07980a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07980a2df0 0x7f07980a52b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005fd0 tx=0x7f07a000af20 comp rx=0 tx=0).stop 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 msgr2=0x7f07b0198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 --2- 172.21.15.107:0/3023490601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07b01082b0 0x7f07b0198730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d010 tx=0x7f07a800bed0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07af7fe700 1 -- 172.21.15.107:0/3023490601 reap_dead start 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 shutdown_connections 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 >> 172.21.15.107:0/3023490601 conn(0x7f07b00780e0 msgr2=0x7f07b00ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 shutdown_connections 2024-04-24T23:43:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.124+0000 7f07b55e1700 1 -- 172.21.15.107:0/3023490601 wait complete. 2024-04-24T23:43:57.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:57 smithi107 ceph-mon[29118]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 2 keys/s, 1 objects/s recovering 2024-04-24T23:43:57.664 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:57 smithi170 ceph-mon[32714]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 2 keys/s, 1 objects/s recovering 2024-04-24T23:43:57.665 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:57 smithi195 ceph-mon[33019]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 13 B/s, 2 keys/s, 1 objects/s recovering 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 -- 172.21.15.107:0/2728203181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b8101b90 msgr2=0x7f48b8101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 --2- 172.21.15.107:0/2728203181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b8101b90 0x7f48b8101f70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f48a80099c0 tx=0x7f48a8009cd0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 -- 172.21.15.107:0/2728203181 shutdown_connections 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 --2- 172.21.15.107:0/2728203181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b81064b0 0x7f48b810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 --2- 172.21.15.107:0/2728203181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b8101b90 0x7f48b8101f70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 --2- 172.21.15.107:0/2728203181 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48b8102540 0x7f48b8105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.975+0000 7f48be40c700 1 -- 172.21.15.107:0/2728203181 >> 172.21.15.107:0/2728203181 conn(0x7f48b80777a0 msgr2=0x7f48b80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.976+0000 7f48be40c700 1 -- 172.21.15.107:0/2728203181 shutdown_connections 2024-04-24T23:43:57.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.976+0000 7f48be40c700 1 -- 172.21.15.107:0/2728203181 wait complete. 2024-04-24T23:43:57.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.977+0000 7f48be40c700 1 Processor -- start 2024-04-24T23:43:57.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.977+0000 7f48be40c700 1 -- start start 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48b8101b90 0x7f48b81a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b8102540 0x7f48b81a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 0x7f48b819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8111540 con 0x7f48b81064b0 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f48b81113c0 con 0x7f48b8102540 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48be40c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f48b81116c0 con 0x7f48b8101b90 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 0x7f48b819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 0x7f48b819ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58660/0 (socket says 172.21.15.107:58660) 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 -- 172.21.15.107:0/819739507 learned_addr learned my addr 172.21.15.107:0/819739507 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48b77fe700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b8102540 0x7f48b81a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48b7fff700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48b8101b90 0x7f48b81a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 -- 172.21.15.107:0/819739507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48b8101b90 msgr2=0x7f48b81a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48b8101b90 0x7f48b81a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 -- 172.21.15.107:0/819739507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b8102540 msgr2=0x7f48b81a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b8102540 0x7f48b81a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48bc9a9700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b819b3f0 con 0x7f48b81064b0 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.978+0000 7f48b77fe700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48b8102540 0x7f48b81a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48bc9a9700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 0x7f48b819ab10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48b000ef30 tx=0x7f48b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b00092e0 con 0x7f48b81064b0 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b0004bb0 con 0x7f48b81064b0 2024-04-24T23:43:57.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b819b680 con 0x7f48b81064b0 2024-04-24T23:43:57.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0004280 con 0x7f48b81064b0 2024-04-24T23:43:57.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.979+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81a77f0 con 0x7f48b81064b0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.980+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48b8062900 con 0x7f48b81064b0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.981+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f48b001b030 con 0x7f48b81064b0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.981+0000 7f48b57fa700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48a00a2cb0 0x7f48a00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b7fff700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48a00a2cb0 0x7f48a00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f48b00cf8e0 con 0x7f48b81064b0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b57fa700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] conn(0x7f48a00ade10 0x7f48a00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48b001fe50 con 0x7f48a00ade10 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f48b00cfd30 con 0x7f48b81064b0 2024-04-24T23:43:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b7fff700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48a00a2cb0 0x7f48a00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48a8000c00 tx=0x7f48a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b77fe700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] conn(0x7f48a00ade10 0x7f48a00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:43:57.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.982+0000 7f48b77fe700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] conn(0x7f48a00ade10 0x7f48a00b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:43:57.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.983+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== osd.0 v2:172.21.15.107:6802/1988153607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f48b001fe50 con 0x7f48a00ade10 2024-04-24T23:43:57.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:57.999+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 --> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f48b8062d80 con 0x7f48a00ade10 2024-04-24T23:43:57.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48b57fa700 1 -- 172.21.15.107:0/819739507 <== osd.0 v2:172.21.15.107:6802/1988153607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f48b8062d80 con 0x7f48a00ade10 2024-04-24T23:43:58.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] conn(0x7f48a00ade10 msgr2=0x7f48a00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:58.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] conn(0x7f48a00ade10 0x7f48a00b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:43:58.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48a00a2cb0 msgr2=0x7f48a00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:58.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48a00a2cb0 0x7f48a00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f48a8000c00 tx=0x7f48a8005fb0 comp rx=0 tx=0).stop 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 msgr2=0x7f48b819ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 --2- 172.21.15.107:0/819739507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48b81064b0 0x7f48b819ab10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48b000ef30 tx=0x7f48b000c3a0 comp rx=0 tx=0).stop 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48bc9a9700 1 -- 172.21.15.107:0/819739507 reap_dead start 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 shutdown_connections 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 >> 172.21.15.107:0/819739507 conn(0x7f48b80777a0 msgr2=0x7f48b8104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 shutdown_connections 2024-04-24T23:43:58.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:43:58.000+0000 7f48be40c700 1 -- 172.21.15.107:0/819739507 wait complete. 2024-04-24T23:43:59.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:43:59 smithi107 ceph-mon[29118]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 11 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:43:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:43:59 smithi195 ceph-mon[33019]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 11 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:43:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:43:59 smithi170 ceph-mon[32714]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 11 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:00.145 INFO:teuthology.orchestra.run.smithi107.stdout:34359738450 2024-04-24T23:44:00.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-24T23:44:00.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 -- 172.21.15.107:0/4019096762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 msgr2=0x7f183010f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 --2- 172.21.15.107:0/4019096762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183010f0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f182800b0a0 tx=0x7f182800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 -- 172.21.15.107:0/4019096762 shutdown_connections 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 --2- 172.21.15.107:0/4019096762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183010f0e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 --2- 172.21.15.107:0/4019096762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18301082a0 0x7f1830108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 --2- 172.21.15.107:0/4019096762 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f183006a7d0 0x7f183006ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 -- 172.21.15.107:0/4019096762 >> 172.21.15.107:0/4019096762 conn(0x7f18300780a0 msgr2=0x7f18300784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 -- 172.21.15.107:0/4019096762 shutdown_connections 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.682+0000 7f1834f1b700 1 -- 172.21.15.107:0/4019096762 wait complete. 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 Processor -- start 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 -- start start 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 0x7f183019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:00.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:00.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18301082a0 0x7f1830198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:00.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18301124f0 con 0x7f183006b170 2024-04-24T23:44:00.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1830112370 con 0x7f183006a7d0 2024-04-24T23:44:00.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.683+0000 7f1834f1b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1830112670 con 0x7f18301082a0 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58676/0 (socket says 172.21.15.107:58676) 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 -- 172.21.15.107:0/1587381693 learned_addr learned my addr 172.21.15.107:0/1587381693 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182ed9d700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18301082a0 0x7f1830198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182e59c700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 0x7f183019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 -- 172.21.15.107:0/1587381693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18301082a0 msgr2=0x7f1830198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18301082a0 0x7f1830198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 -- 172.21.15.107:0/1587381693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 msgr2=0x7f183019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 0x7f183019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 -- 172.21.15.107:0/1587381693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1830198ff0 con 0x7f183006b170 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182e59c700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 0x7f183019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f182dd9b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f182400d950 tx=0x7f182400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:00.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824016070 con 0x7f183006b170 2024-04-24T23:44:00.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1824009e20 con 0x7f183006b170 2024-04-24T23:44:00.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18301992e0 con 0x7f183006b170 2024-04-24T23:44:00.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182400f620 con 0x7f183006b170 2024-04-24T23:44:00.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.684+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18301a55c0 con 0x7f183006b170 2024-04-24T23:44:00.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.685+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1830103950 con 0x7f183006b170 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.687+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1824019070 con 0x7f183006b170 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.687+0000 7f181f7fe700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 0x7f18180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.687+0000 7f182e59c700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 0x7f18180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.687+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f18240cf0f0 con 0x7f183006b170 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.688+0000 7f182e59c700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 0x7f18180a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1820005fd0 tx=0x7f182000a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:00.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.691+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f182409bb90 con 0x7f183006b170 2024-04-24T23:44:00.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.893+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f18301a5790 con 0x7f183006b170 2024-04-24T23:44:00.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.894+0000 7f181f7fe700 1 -- 172.21.15.107:0/1587381693 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f182409b490 con 0x7f183006b170 2024-04-24T23:44:00.894 INFO:teuthology.orchestra.run.smithi107.stdout:77309411402 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 msgr2=0x7f18180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 0x7f18180a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1820005fd0 tx=0x7f182000a5e0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 msgr2=0x7f183019eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f182400d950 tx=0x7f182400dc60 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 shutdown_connections 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f18180a2cb0 0x7f18180a5170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f18301082a0 0x7f1830198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f183006a7d0 0x7f183019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 --2- 172.21.15.107:0/1587381693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f183006b170 0x7f183019eb80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 >> 172.21.15.107:0/1587381693 conn(0x7f18300780a0 msgr2=0x7f18300ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 shutdown_connections 2024-04-24T23:44:00.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:00.896+0000 7f1834f1b700 1 -- 172.21.15.107:0/1587381693 wait complete. 2024-04-24T23:44:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:01 smithi195 ceph-mon[33019]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 10 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:01 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1587381693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T23:44:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:01 smithi170 ceph-mon[32714]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 10 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:01 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1587381693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T23:44:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:01 smithi107 ceph-mon[29118]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 10 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1587381693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T23:44:01.757 INFO:teuthology.orchestra.run.smithi107.stdout:219043332143 2024-04-24T23:44:01.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- 172.21.15.107:0/347968324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350101800 msgr2=0x7fe350106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 --2- 172.21.15.107:0/347968324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350101800 0x7fe350106f40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe3400099c0 tx=0x7fe340009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- 172.21.15.107:0/347968324 shutdown_connections 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 --2- 172.21.15.107:0/347968324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350107670 0x7fe35010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 --2- 172.21.15.107:0/347968324 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350100e50 0x7fe350101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 --2- 172.21.15.107:0/347968324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350101800 0x7fe350106f40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- 172.21.15.107:0/347968324 >> 172.21.15.107:0/347968324 conn(0x7fe350076450 msgr2=0x7fe350076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- 172.21.15.107:0/347968324 shutdown_connections 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- 172.21.15.107:0/347968324 wait complete. 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 Processor -- start 2024-04-24T23:44:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.396+0000 7fe357b04700 1 -- start start 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350101800 0x7fe35019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350107670 0x7fe3501988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe350112660 con 0x7fe350100e50 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501124e0 con 0x7fe350101800 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501127e0 con 0x7fe350107670 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58692/0 (socket says 172.21.15.107:58692) 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 -- 172.21.15.107:0/29275554 learned_addr learned my addr 172.21.15.107:0/29275554 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe35509f700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350101800 0x7fe35019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3560a1700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350107670 0x7fe3501988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 -- 172.21.15.107:0/29275554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350107670 msgr2=0x7fe3501988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350107670 0x7fe3501988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 -- 172.21.15.107:0/29275554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350101800 msgr2=0x7fe35019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350101800 0x7fe35019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 -- 172.21.15.107:0/29275554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe350199040 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe3558a0700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00b530 tx=0x7fe34c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c015020 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe34c011d10 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe350199330 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00f2e0 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.397+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3501a55f0 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.398+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe35010a820 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.399+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe34c018020 con 0x7fe350100e50 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.400+0000 7fe346ffd700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 0x7fe33c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.400+0000 7fe35509f700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 0x7fe33c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.400+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe34c0ceb40 con 0x7fe350100e50 2024-04-24T23:44:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.400+0000 7fe35509f700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 0x7fe33c0a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe340000c00 tx=0x7fe340005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.404+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe34c01f020 con 0x7fe350100e50 2024-04-24T23:44:02.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.608+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe350199e10 con 0x7fe350100e50 2024-04-24T23:44:02.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.608+0000 7fe346ffd700 1 -- 172.21.15.107:0/29275554 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe34c09b6f0 con 0x7fe350100e50 2024-04-24T23:44:02.609 INFO:teuthology.orchestra.run.smithi107.stdout:141733920832 2024-04-24T23:44:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.610+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 msgr2=0x7fe33c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.610+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 0x7fe33c0a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe340000c00 tx=0x7fe340005fb0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.610+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 msgr2=0x7fe35019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.610+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00b530 tx=0x7fe34c00b840 comp rx=0 tx=0).stop 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 shutdown_connections 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe33c0a2cb0 0x7fe33c0a5170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe350107670 0x7fe3501988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe350100e50 0x7fe35019e720 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 --2- 172.21.15.107:0/29275554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe350101800 0x7fe35019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 >> 172.21.15.107:0/29275554 conn(0x7fe350076450 msgr2=0x7fe350100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 shutdown_connections 2024-04-24T23:44:02.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:02.611+0000 7fe357b04700 1 -- 172.21.15.107:0/29275554 wait complete. 2024-04-24T23:44:03.295 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920832 for osd.4 2024-04-24T23:44:03.295 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:03 smithi107 ceph-mon[29118]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:03.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:03 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/29275554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T23:44:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:03 smithi195 ceph-mon[33019]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:03 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/29275554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T23:44:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:03 smithi170 ceph-mon[32714]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-04-24T23:44:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:03 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/29275554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T23:44:04.281 INFO:teuthology.orchestra.run.smithi107.stdout:107374182467 2024-04-24T23:44:04.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-24T23:44:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 -- 172.21.15.107:0/3008315416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f101013c0 msgr2=0x7f9f101017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3008315416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f101013c0 0x7f9f101017a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9f000099c0 tx=0x7f9f00009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 -- 172.21.15.107:0/3008315416 shutdown_connections 2024-04-24T23:44:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3008315416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 0x7f9f1010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3008315416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f101013c0 0x7f9f101017a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3008315416 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f10101d70 0x7f9f10105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 -- 172.21.15.107:0/3008315416 >> 172.21.15.107:0/3008315416 conn(0x7f9f10078100 msgr2=0x7f9f10078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 -- 172.21.15.107:0/3008315416 shutdown_connections 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.152+0000 7f9f153f7700 1 -- 172.21.15.107:0/3008315416 wait complete. 2024-04-24T23:44:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 Processor -- start 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 -- start start 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f101013c0 0x7f9f10102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 0x7f9f10103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 0x7f9f10103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f10111430 con 0x7f9f10101d70 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9f101112b0 con 0x7f9f10106450 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f153f7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f9f101115b0 con 0x7f9f101013c0 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f0e7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 0x7f9f10103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f0e7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 0x7f9f10103350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58714/0 (socket says 172.21.15.107:58714) 2024-04-24T23:44:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.153+0000 7f9f0e7fc700 1 -- 172.21.15.107:0/3516146886 learned_addr learned my addr 172.21.15.107:0/3516146886 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0effd700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f101013c0 0x7f9f10102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0f7fe700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 0x7f9f10103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 -- 172.21.15.107:0/3516146886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f101013c0 msgr2=0x7f9f10102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f101013c0 0x7f9f10102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 -- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 msgr2=0x7f9f10103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 0x7f9f10103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f101a6f80 con 0x7f9f10101d70 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0effd700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9f101013c0 0x7f9f10102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0f7fe700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f10106450 0x7f9f10103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f0e7fc700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 0x7f9f10103350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400e970 tx=0x7f9f0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0400cbd0 con 0x7f9f10101d70 2024-04-24T23:44:05.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f101a7270 con 0x7f9f10101d70 2024-04-24T23:44:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0400cd70 con 0x7f9f10101d70 2024-04-24T23:44:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04010430 con 0x7f9f10101d70 2024-04-24T23:44:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.154+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f101a7800 con 0x7f9f10101d70 2024-04-24T23:44:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.155+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f10062940 con 0x7f9f10101d70 2024-04-24T23:44:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.156+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9f04004750 con 0x7f9f10101d70 2024-04-24T23:44:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9ef7fff700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9ef80a2cb0 0x7f9ef80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9f0effd700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9ef80a2cb0 0x7f9ef80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f040cfcf0 con 0x7f9f10101d70 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9ef7fff700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] conn(0x7f9ef80ade00 0x7f9ef80b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ef80b0970 con 0x7f9ef80ade00 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9f040d0140 con 0x7f9f10101d70 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9f0effd700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9ef80a2cb0 0x7f9ef80a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9f00000c00 tx=0x7f9f00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.157+0000 7f9f0f7fe700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] conn(0x7f9ef80ade00 0x7f9ef80b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.158+0000 7f9f0f7fe700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] conn(0x7f9ef80ade00 0x7f9ef80b02a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.159+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== osd.10 v2:172.21.15.170:6824/1507442612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ef80b0970 con 0x7f9ef80ade00 2024-04-24T23:44:05.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.175+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 --> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f10062d40 con 0x7f9ef80ade00 2024-04-24T23:44:05.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.175+0000 7f9ef7fff700 1 -- 172.21.15.107:0/3516146886 <== osd.10 v2:172.21.15.170:6824/1507442612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9f10062d40 con 0x7f9ef80ade00 2024-04-24T23:44:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] conn(0x7f9ef80ade00 msgr2=0x7f9ef80b02a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] conn(0x7f9ef80ade00 0x7f9ef80b02a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9ef80a2cb0 msgr2=0x7f9ef80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9ef80a2cb0 0x7f9ef80a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9f00000c00 tx=0x7f9f00005fd0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 msgr2=0x7f9f10103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 --2- 172.21.15.107:0/3516146886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f10101d70 0x7f9f10103350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400e970 tx=0x7f9f0400ed30 comp rx=0 tx=0).stop 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f0f7fe700 1 -- 172.21.15.107:0/3516146886 reap_dead start 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 shutdown_connections 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 >> 172.21.15.107:0/3516146886 conn(0x7f9f10078100 msgr2=0x7f9f1006ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 shutdown_connections 2024-04-24T23:44:05.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.176+0000 7f9f153f7700 1 -- 172.21.15.107:0/3516146886 wait complete. 2024-04-24T23:44:05.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:05 smithi107 ceph-mon[29118]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:05.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411402 for osd.2 2024-04-24T23:44:05.539 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:05 smithi195 ceph-mon[33019]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:05 smithi170 ceph-mon[32714]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:05.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 -- 172.21.15.107:0/1619659728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c06a480 msgr2=0x7fee3c06a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 --2- 172.21.15.107:0/1619659728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c06a480 0x7fee3c06a8e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009990 tx=0x7fee2c009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 -- 172.21.15.107:0/1619659728 shutdown_connections 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 --2- 172.21.15.107:0/1619659728 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c104c70 0x7fee3c105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 --2- 172.21.15.107:0/1619659728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 0x7fee3c069f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 --2- 172.21.15.107:0/1619659728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c06a480 0x7fee3c06a8e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 -- 172.21.15.107:0/1619659728 >> 172.21.15.107:0/1619659728 conn(0x7fee3c0780e0 msgr2=0x7fee3c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 -- 172.21.15.107:0/1619659728 shutdown_connections 2024-04-24T23:44:05.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.951+0000 7fee43fba700 1 -- 172.21.15.107:0/1619659728 wait complete. 2024-04-24T23:44:05.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 Processor -- start 2024-04-24T23:44:05.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 -- start start 2024-04-24T23:44:05.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c06a480 0x7fee3c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 0x7fee3c1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c118af0 con 0x7fee3c104c70 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c118970 con 0x7fee3c107160 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee43fba700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c118c70 con 0x7fee3c06a480 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee41555700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee41555700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58734/0 (socket says 172.21.15.107:58734) 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.952+0000 7fee41555700 1 -- 172.21.15.107:0/2861570560 learned_addr learned my addr 172.21.15.107:0/2861570560 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41d56700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c06a480 0x7fee3c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee42557700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 0x7fee3c1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 -- 172.21.15.107:0/2861570560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c06a480 msgr2=0x7fee3c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c06a480 0x7fee3c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 -- 172.21.15.107:0/2861570560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 msgr2=0x7fee3c1a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 0x7fee3c1a3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:05.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 -- 172.21.15.107:0/2861570560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c1a3fb0 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee41555700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c027070 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2c004190 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00f460 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3c1a4240 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.953+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3c1abb20 con 0x7fee3c104c70 2024-04-24T23:44:05.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.954+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3c068f20 con 0x7fee3c104c70 2024-04-24T23:44:05.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.955+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fee2c02a070 con 0x7fee3c104c70 2024-04-24T23:44:05.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.956+0000 7fee32ffd700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 0x7fee280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:05.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.956+0000 7fee41d56700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 0x7fee280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:05.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.956+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fee2c0d0300 con 0x7fee3c104c70 2024-04-24T23:44:05.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.957+0000 7fee41d56700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 0x7fee280a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee38005fd0 tx=0x7fee38005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:05.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:05.963+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee2c013070 con 0x7fee3c104c70 2024-04-24T23:44:06.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.168+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fee3c1abcf0 con 0x7fee3c104c70 2024-04-24T23:44:06.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.169+0000 7fee32ffd700 1 -- 172.21.15.107:0/2861570560 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fee2c09ced0 con 0x7fee3c104c70 2024-04-24T23:44:06.169 INFO:teuthology.orchestra.run.smithi107.stdout:253403070507 2024-04-24T23:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 msgr2=0x7fee280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 0x7fee280a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee38005fd0 tx=0x7fee38005db0 comp rx=0 tx=0).stop 2024-04-24T23:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 msgr2=0x7fee3c19f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c003730 comp rx=0 tx=0).stop 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 shutdown_connections 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fee280a2cb0 0x7fee280a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee3c107160 0x7fee3c1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fee3c06a480 0x7fee3c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 --2- 172.21.15.107:0/2861570560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee3c104c70 0x7fee3c19f4e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 >> 172.21.15.107:0/2861570560 conn(0x7fee3c0780e0 msgr2=0x7fee3c103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 shutdown_connections 2024-04-24T23:44:06.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:06.171+0000 7fee43fba700 1 -- 172.21.15.107:0/2861570560 wait complete. 2024-04-24T23:44:06.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2861570560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T23:44:06.462 INFO:teuthology.orchestra.run.smithi107.stdout:309237645345 2024-04-24T23:44:06.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-24T23:44:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2861570560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T23:44:06.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2861570560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T23:44:07.072 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:07.075 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:07.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:07 smithi107 ceph-mon[29118]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:07.504 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070507 for osd.8 2024-04-24T23:44:07.504 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:07.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:07 smithi170 ceph-mon[32714]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:07 smithi195 ceph-mon[33019]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.202+0000 7f8d20a09700 1 -- 172.21.15.107:0/3127765634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074d80 msgr2=0x7f8d1c10e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.202+0000 7f8d20a09700 1 --2- 172.21.15.107:0/3127765634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074d80 0x7f8d1c10e720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009990 tx=0x7f8d10009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 -- 172.21.15.107:0/3127765634 shutdown_connections 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 --2- 172.21.15.107:0/3127765634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c10ec60 0x7f8d1c111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 --2- 172.21.15.107:0/3127765634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d1c074460 0x7f8d1c074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 --2- 172.21.15.107:0/3127765634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074d80 0x7f8d1c10e720 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 -- 172.21.15.107:0/3127765634 >> 172.21.15.107:0/3127765634 conn(0x7f8d1c0793e0 msgr2=0x7f8d1c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:08.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 -- 172.21.15.107:0/3127765634 shutdown_connections 2024-04-24T23:44:08.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.203+0000 7f8d20a09700 1 -- 172.21.15.107:0/3127765634 wait complete. 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.204+0000 7f8d20a09700 1 Processor -- start 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.204+0000 7f8d20a09700 1 -- start start 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074460 0x7f8d1c102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 0x7f8d1c1028c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d1c10ec60 0x7f8d1c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c114770 con 0x7f8d1c074460 2024-04-24T23:44:08.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1145f0 con 0x7f8d1c074d80 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d20a09700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c1148f0 con 0x7f8d1c10ec60 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 0x7f8d1c1028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d1ad9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d1c10ec60 0x7f8d1c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 0x7f8d1c1028c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:46078/0 (socket says 172.21.15.107:46078) 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 -- 172.21.15.107:0/205064161 learned_addr learned my addr 172.21.15.107:0/205064161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d1a59c700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074460 0x7f8d1c102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 -- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d1c10ec60 msgr2=0x7f8d1c102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d1c10ec60 0x7f8d1c102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 -- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074460 msgr2=0x7f8d1c102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074460 0x7f8d1c102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d19d9b700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c1ab290 con 0x7f8d1c074d80 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.205+0000 7f8d1a59c700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d1c074460 0x7f8d1c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:08.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d19d9b700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 0x7f8d1c1028c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d10000c00 tx=0x7f8d1000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10027070 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d10003d70 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000f460 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c1ab520 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.206+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c1ab8c0 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.207+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8d1c0629c0 con 0x7f8d1c074d80 2024-04-24T23:44:08.208 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:08.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.209+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8d1002a070 con 0x7f8d1c074d80 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.209+0000 7f8d0b7fe700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d040a2ce0 0x7f8d040a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.210+0000 7f8d1a59c700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d040a2ce0 0x7f8d040a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.210+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d100d0490 con 0x7f8d1c074d80 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.210+0000 7f8d0b7fe700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] conn(0x7f8d040ade40 0x7f8d040b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.210+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d10001d30 con 0x7f8d040ade40 2024-04-24T23:44:08.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.210+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8d100d7050 con 0x7f8d1c074d80 2024-04-24T23:44:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.211+0000 7f8d1a59c700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d040a2ce0 0x7f8d040a51a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009cb0 tx=0x7f8d0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.211+0000 7f8d1ad9d700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] conn(0x7f8d040ade40 0x7f8d040b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:08.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.211+0000 7f8d1ad9d700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] conn(0x7f8d040ade40 0x7f8d040b02e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:08.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.212+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== osd.11 v2:172.21.15.195:6824/2390923874 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8d10001d30 con 0x7f8d040ade40 2024-04-24T23:44:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.241+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 --> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8d1c062e70 con 0x7f8d040ade40 2024-04-24T23:44:08.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d0b7fe700 1 -- 172.21.15.107:0/205064161 <== osd.11 v2:172.21.15.195:6824/2390923874 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8d1c062e70 con 0x7f8d040ade40 2024-04-24T23:44:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] conn(0x7f8d040ade40 msgr2=0x7f8d040b02e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] conn(0x7f8d040ade40 0x7f8d040b02e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d040a2ce0 msgr2=0x7f8d040a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8d040a2ce0 0x7f8d040a51a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009cb0 tx=0x7f8d0c009260 comp rx=0 tx=0).stop 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 msgr2=0x7f8d1c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 --2- 172.21.15.107:0/205064161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c074d80 0x7f8d1c1028c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d10000c00 tx=0x7f8d1000bab0 comp rx=0 tx=0).stop 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d1ad9d700 1 -- 172.21.15.107:0/205064161 reap_dead start 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 shutdown_connections 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 >> 172.21.15.107:0/205064161 conn(0x7f8d1c0793e0 msgr2=0x7f8d1c0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 shutdown_connections 2024-04-24T23:44:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:08.242+0000 7f8d20a09700 1 -- 172.21.15.107:0/205064161 wait complete. 2024-04-24T23:44:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:09.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:09 smithi107 ceph-mon[29118]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:09 smithi195 ceph-mon[33019]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:09 smithi170 ceph-mon[32714]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:10.781 INFO:teuthology.orchestra.run.smithi107.stdout:343597383708 2024-04-24T23:44:10.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-24T23:44:11.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:11 smithi107 ceph-mon[29118]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:11 smithi195 ceph-mon[33019]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:11 smithi170 ceph-mon[32714]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:13 smithi195 ceph-mon[33019]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:13 smithi170 ceph-mon[32714]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:13.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:13 smithi107 ceph-mon[29118]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:15.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.406+0000 7fb4c3c77700 1 -- 172.21.15.107:0/2300545625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 msgr2=0x7fb4bc101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:15.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.406+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/2300545625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc101230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80099c0 tx=0x7fb4b8009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 -- 172.21.15.107:0/2300545625 shutdown_connections 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/2300545625 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc107670 0x7fb4bc10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/2300545625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc101230 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/2300545625 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc101800 0x7fb4bc106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 -- 172.21.15.107:0/2300545625 >> 172.21.15.107:0/2300545625 conn(0x7fb4bc076450 msgr2=0x7fb4bc076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 -- 172.21.15.107:0/2300545625 shutdown_connections 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 -- 172.21.15.107:0/2300545625 wait complete. 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 Processor -- start 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.407+0000 7fb4c3c77700 1 -- start start 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 0x7fb4bc19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 0x7fb4bc1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc112660 con 0x7fb4bc100e50 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc1124e0 con 0x7fb4bc101800 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb4bc1127e0 con 0x7fb4bc107670 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39008/0 (socket says 172.21.15.107:39008) 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 -- 172.21.15.107:0/4156847855 learned_addr learned my addr 172.21.15.107:0/4156847855 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1212700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 0x7fb4bc19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c2214700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 0x7fb4bc1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 -- 172.21.15.107:0/4156847855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 msgr2=0x7fb4bc1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 0x7fb4bc1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 -- 172.21.15.107:0/4156847855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 msgr2=0x7fb4bc19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 0x7fb4bc19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 -- 172.21.15.107:0/4156847855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4bc199040 con 0x7fb4bc100e50 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1212700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 0x7fb4bc19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c2214700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 0x7fb4bc1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c1a13700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009fd0 tx=0x7fb4b8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8027070 con 0x7fb4bc100e50 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4bc1992d0 con 0x7fb4bc100e50 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4b8004150 con 0x7fb4bc100e50 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.408+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800f460 con 0x7fb4bc100e50 2024-04-24T23:44:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.409+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4bc101d00 con 0x7fb4bc100e50 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.410+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a0005330 con 0x7fb4bc100e50 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.411+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb4b802a070 con 0x7fb4bc100e50 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.411+0000 7fb4b2ffd700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 0x7fb4a80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.411+0000 7fb4c1212700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 0x7fb4a80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.411+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb4b80d0420 con 0x7fb4bc100e50 2024-04-24T23:44:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.412+0000 7fb4c1212700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 0x7fb4a80a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc108b50 tx=0x7fb4ac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.420+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb4b805e740 con 0x7fb4bc100e50 2024-04-24T23:44:15.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:15 smithi107 ceph-mon[29118]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:15 smithi195 ceph-mon[33019]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:15 smithi170 ceph-mon[32714]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.623+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb4a00058f0 con 0x7fb4bc100e50 2024-04-24T23:44:15.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.623+0000 7fb4b2ffd700 1 -- 172.21.15.107:0/4156847855 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb4b8013070 con 0x7fb4bc100e50 2024-04-24T23:44:15.623 INFO:teuthology.orchestra.run.smithi107.stdout:219043332146 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.625+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 msgr2=0x7fb4a80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.625+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 0x7fb4a80a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc108b50 tx=0x7fb4ac008040 comp rx=0 tx=0).stop 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.625+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 msgr2=0x7fb4bc19e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.625+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009fd0 tx=0x7fb4b8003730 comp rx=0 tx=0).stop 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 shutdown_connections 2024-04-24T23:44:15.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb4a80a2d30 0x7fb4a80a51f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb4bc107670 0x7fb4bc1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4bc100e50 0x7fb4bc19e720 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 --2- 172.21.15.107:0/4156847855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb4bc101800 0x7fb4bc19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 >> 172.21.15.107:0/4156847855 conn(0x7fb4bc076450 msgr2=0x7fb4bc100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 shutdown_connections 2024-04-24T23:44:15.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:15.626+0000 7fb4c3c77700 1 -- 172.21.15.107:0/4156847855 wait complete. 2024-04-24T23:44:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4156847855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T23:44:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:16 smithi195 ceph-mon[33019]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4156847855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T23:44:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:16 smithi170 ceph-mon[32714]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:16.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4156847855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T23:44:16.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:16 smithi107 ceph-mon[29118]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:17.143 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332143 got 219043332146 for osd.7 2024-04-24T23:44:17.143 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:17 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:17 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:17 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:17 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:17 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:17 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/4270004073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac1082c0 msgr2=0x7fc7ac1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/4270004073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac1086a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b0a0 tx=0x7fc7a400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/4270004073 shutdown_connections 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/4270004073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 0x7fc7ac10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/4270004073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac1086a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/4270004073 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac100820 0x7fc7ac100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/4270004073 >> 172.21.15.107:0/4270004073 conn(0x7fc7ac0777a0 msgr2=0x7fc7ac0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/4270004073 shutdown_connections 2024-04-24T23:44:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/4270004073 wait complete. 2024-04-24T23:44:17.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.917+0000 7fc7b1dc7700 1 Processor -- start 2024-04-24T23:44:17.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 -- start start 2024-04-24T23:44:17.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:17.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 0x7fc7ac19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:17.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac112580 con 0x7fc7ac100820 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac112400 con 0x7fc7ac1011c0 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7b1dc7700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac112700 con 0x7fc7ac1082c0 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45392/0 (socket says 172.21.15.107:45392) 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 -- 172.21.15.107:0/1544925500 learned_addr learned my addr 172.21.15.107:0/1544925500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7aaffd700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 0x7fc7ac19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7abfff700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 -- 172.21.15.107:0/1544925500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac1082c0 msgr2=0x7fc7ac198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 -- 172.21.15.107:0/1544925500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 msgr2=0x7fc7ac19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 0x7fc7ac19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.918+0000 7fc7ab7fe700 1 -- 172.21.15.107:0/1544925500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac198fe0 con 0x7fc7ac100820 2024-04-24T23:44:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7ab7fe700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4000f80 tx=0x7fc7a400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4021070 con 0x7fc7ac100820 2024-04-24T23:44:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac199270 con 0x7fc7ac100820 2024-04-24T23:44:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a4007db0 con 0x7fc7ac100820 2024-04-24T23:44:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4005550 con 0x7fc7ac100820 2024-04-24T23:44:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.919+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ac101930 con 0x7fc7ac100820 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.920+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ac068f20 con 0x7fc7ac100820 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.921+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc7a4024070 con 0x7fc7ac100820 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.922+0000 7fc7a8ff9700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 0x7fc7980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.922+0000 7fc7aaffd700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 0x7fc7980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.922+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc7a400f020 con 0x7fc7ac100820 2024-04-24T23:44:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.922+0000 7fc7aaffd700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 0x7fc7980a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc794006fd0 tx=0x7fc794008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.929+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc7a402b030 con 0x7fc7ac100820 2024-04-24T23:44:18.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.148+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc7ac02db30 con 0x7fc7ac100820 2024-04-24T23:44:18.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.149+0000 7fc7a8ff9700 1 -- 172.21.15.107:0/1544925500 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc7a402ee20 con 0x7fc7ac100820 2024-04-24T23:44:18.149 INFO:teuthology.orchestra.run.smithi107.stdout:163208757309 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 msgr2=0x7fc7980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 0x7fc7980a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc794006fd0 tx=0x7fc794008040 comp rx=0 tx=0).stop 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 msgr2=0x7fc7ac19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4000f80 tx=0x7fc7a400bf10 comp rx=0 tx=0).stop 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 shutdown_connections 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7980a2cb0 0x7fc7980a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc7ac1082c0 0x7fc7ac198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7ac100820 0x7fc7ac19e680 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 --2- 172.21.15.107:0/1544925500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac1011c0 0x7fc7ac19ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 >> 172.21.15.107:0/1544925500 conn(0x7fc7ac0777a0 msgr2=0x7fc7ac0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 shutdown_connections 2024-04-24T23:44:18.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.151+0000 7fc7b1dc7700 1 -- 172.21.15.107:0/1544925500 wait complete. 2024-04-24T23:44:18.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2288933476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T23:44:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:44:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:44:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:18.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1544925500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2288933476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1544925500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T23:44:18.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 -- 172.21.15.107:0/1274579231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc4101130 msgr2=0x7f9bc41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/1274579231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc4101130 0x7f9bc41015b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009990 tx=0x7f9bb8009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 -- 172.21.15.107:0/1274579231 shutdown_connections 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/1274579231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101af0 0x7f9bc410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/1274579231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc41071a0 0x7f9bc4107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/1274579231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc4101130 0x7f9bc41015b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 -- 172.21.15.107:0/1274579231 >> 172.21.15.107:0/1274579231 conn(0x7f9bc40780e0 msgr2=0x7f9bc40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 -- 172.21.15.107:0/1274579231 shutdown_connections 2024-04-24T23:44:18.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 -- 172.21.15.107:0/1274579231 wait complete. 2024-04-24T23:44:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.610+0000 7f9bc99c1700 1 Processor -- start 2024-04-24T23:44:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 -- start start 2024-04-24T23:44:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101130 0x7f9bc4199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc4101af0 0x7f9bc4199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:18.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4111490 con 0x7f9bc41071a0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4111310 con 0x7f9bc4101130 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc99c1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4111610 con 0x7f9bc4101af0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45424/0 (socket says 172.21.15.107:45424) 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 -- 172.21.15.107:0/133451898 learned_addr learned my addr 172.21.15.107:0/133451898 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc2ffd700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101130 0x7f9bc4199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc27fc700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc4101af0 0x7f9bc4199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 -- 172.21.15.107:0/133451898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc4101af0 msgr2=0x7f9bc4199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc4101af0 0x7f9bc4199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 -- 172.21.15.107:0/133451898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101130 msgr2=0x7f9bc4199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101130 0x7f9bc4199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.611+0000 7f9bc37fe700 1 -- 172.21.15.107:0/133451898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc41a4e50 con 0x7f9bc41071a0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc37fe700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc007ee0 tx=0x7f9bbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:18.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00ddd0 con 0x7f9bc41071a0 2024-04-24T23:44:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bbc004020 con 0x7f9bc41071a0 2024-04-24T23:44:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc015660 con 0x7f9bc41071a0 2024-04-24T23:44:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc41a5080 con 0x7f9bc41071a0 2024-04-24T23:44:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.612+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc41a5570 con 0x7f9bc41071a0 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.613+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc4109710 con 0x7f9bc41071a0 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.614+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9bbc0041d0 con 0x7f9bc41071a0 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.615+0000 7f9bc89bf700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 0x7f9bac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.615+0000 7f9bc2ffd700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 0x7f9bac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.615+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9bbc0cf790 con 0x7f9bc41071a0 2024-04-24T23:44:18.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.615+0000 7f9bc2ffd700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 0x7f9bac0a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009a10 tx=0x7f9bb4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:18.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.618+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9bbc010070 con 0x7f9bc41071a0 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2288933476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:44:18.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1544925500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T23:44:18.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.820+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9bc41090f0 con 0x7f9bc41071a0 2024-04-24T23:44:18.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.821+0000 7f9bc89bf700 1 -- 172.21.15.107:0/133451898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9bbc09c230 con 0x7f9bc41071a0 2024-04-24T23:44:18.821 INFO:teuthology.orchestra.run.smithi107.stdout:34359738454 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 msgr2=0x7f9bac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 0x7f9bac0a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009a10 tx=0x7f9bb4009190 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 msgr2=0x7f9bc41a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc007ee0 tx=0x7f9bbc00d3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 shutdown_connections 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9bac0a2cb0 0x7f9bac0a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc41071a0 0x7f9bc41a4910 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bc4101130 0x7f9bc4199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 --2- 172.21.15.107:0/133451898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9bc4101af0 0x7f9bc4199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 >> 172.21.15.107:0/133451898 conn(0x7f9bc40780e0 msgr2=0x7f9bc40ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 shutdown_connections 2024-04-24T23:44:18.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.823+0000 7f9bc99c1700 1 -- 172.21.15.107:0/133451898 wait complete. 2024-04-24T23:44:19.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 -- 172.21.15.107:0/1093935837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c107540 msgr2=0x7f118c10f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 --2- 172.21.15.107:0/1093935837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c107540 0x7f118c10f110 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f118400b0d0 tx=0x7f118400b3e0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 -- 172.21.15.107:0/1093935837 shutdown_connections 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 --2- 172.21.15.107:0/1093935837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c107540 0x7f118c10f110 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 --2- 172.21.15.107:0/1093935837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c102550 0x7f118c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 --2- 172.21.15.107:0/1093935837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 0x7f118c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 -- 172.21.15.107:0/1093935837 >> 172.21.15.107:0/1093935837 conn(0x7f118c0793f0 msgr2=0x7f118c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 -- 172.21.15.107:0/1093935837 shutdown_connections 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.328+0000 7f119298f700 1 -- 172.21.15.107:0/1093935837 wait complete. 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 Processor -- start 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 -- start start 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:19.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 0x7f118c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c107540 0x7f118c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f118c112550 con 0x7f118c102550 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1123d0 con 0x7f118c102f00 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.329+0000 7f119298f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1126d0 con 0x7f118c107540 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45388/0 (socket says 172.21.15.107:45388) 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 -- 172.21.15.107:0/2288933476 learned_addr learned my addr 172.21.15.107:0/2288933476 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f1190f2c700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c107540 0x7f118c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:19.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118b7fe700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 0x7f118c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stdout:309237645348 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 -- 172.21.15.107:0/2288933476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c107540 msgr2=0x7f118c074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c107540 0x7f118c074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 -- 172.21.15.107:0/2288933476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 msgr2=0x7f118c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 0x7f118c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 -- 172.21.15.107:0/2288933476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c1a6f30 con 0x7f118c102550 2024-04-24T23:44:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f118bfff700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f117c00c970 tx=0x7f117c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.330+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c007aa0 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.331+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118c1a7220 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.331+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118c1a7760 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.331+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f117c004d10 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.331+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c0056a0 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.333+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118c068f20 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.333+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f117c00f490 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.334+0000 7f11897fa700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 0x7f11740a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.334+0000 7f118b7fe700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 0x7f11740a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.334+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f117c0cf2b0 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.334+0000 7f118b7fe700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 0x7f11740a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1180005fd0 tx=0x7f1180005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.338+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f117c09ce00 con 0x7f118c102550 2024-04-24T23:44:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.552+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f118c02db30 con 0x7f118c102550 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.552+0000 7f11897fa700 1 -- 172.21.15.107:0/2288933476 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f117c09c700 con 0x7f118c102550 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 msgr2=0x7f11740a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 0x7f11740a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1180005fd0 tx=0x7f1180005db0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 msgr2=0x7f118c074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f117c00c970 tx=0x7f117c00cc80 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 shutdown_connections 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f11740a2d30 0x7f11740a51f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f118c107540 0x7f118c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f118c102550 0x7f118c074530 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 --2- 172.21.15.107:0/2288933476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c102f00 0x7f118c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 >> 172.21.15.107:0/2288933476 conn(0x7f118c0793f0 msgr2=0x7f118c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 shutdown_connections 2024-04-24T23:44:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:17.555+0000 7f119298f700 1 -- 172.21.15.107:0/2288933476 wait complete. 2024-04-24T23:44:19.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1396330634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T23:44:19.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/133451898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T23:44:19.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1396330634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T23:44:19.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/133451898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T23:44:19.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1396330634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T23:44:19.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/133451898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T23:44:19.986 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645348 for osd.10 2024-04-24T23:44:19.987 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:20.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 -- 172.21.15.107:0/3579476669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102ec0 msgr2=0x7f03c8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3579476669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102ec0 0x7f03c8106f00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0099c0 tx=0x7f03bc009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 -- 172.21.15.107:0/3579476669 shutdown_connections 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3579476669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 0x7f03c810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3579476669 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102510 0x7f03c81028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3579476669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102ec0 0x7f03c8106f00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 -- 172.21.15.107:0/3579476669 >> 172.21.15.107:0/3579476669 conn(0x7f03c80793e0 msgr2=0x7f03c80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 -- 172.21.15.107:0/3579476669 shutdown_connections 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.179+0000 7f03ce52c700 1 -- 172.21.15.107:0/3579476669 wait complete. 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 Processor -- start 2024-04-24T23:44:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 -- start start 2024-04-24T23:44:20.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102ec0 0x7f03c819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 0x7f03c8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03c81125a0 con 0x7f03c8102510 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f03c8112420 con 0x7f03c81075e0 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03ce52c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f03c8112720 con 0x7f03c8102ec0 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03c7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03c7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45444/0 (socket says 172.21.15.107:45444) 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.180+0000 7f03c7fff700 1 -- 172.21.15.107:0/3082217731 learned_addr learned my addr 172.21.15.107:0/3082217731 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03ccac9700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 0x7f03c8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c77fe700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102ec0 0x7f03c819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 -- 172.21.15.107:0/3082217731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102ec0 msgr2=0x7f03c819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102ec0 0x7f03c819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 -- 172.21.15.107:0/3082217731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 msgr2=0x7f03c8198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 0x7f03c8198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 -- 172.21.15.107:0/3082217731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c81990c0 con 0x7f03c8102510 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c7fff700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f03b800d950 tx=0x7f03b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8016070 con 0x7f03c8102510 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c81993b0 con 0x7f03c8102510 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b800fd70 con 0x7f03c8102510 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b800e610 con 0x7f03c8102510 2024-04-24T23:44:20.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.181+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c81a5620 con 0x7f03c8102510 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.182+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c8068f20 con 0x7f03c8102510 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.183+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f03b8019070 con 0x7f03c8102510 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.184+0000 7f03c57fa700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 0x7f03b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.184+0000 7f03c77fe700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 0x7f03b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.184+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f03b80cefc0 con 0x7f03c8102510 2024-04-24T23:44:20.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.184+0000 7f03c77fe700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 0x7f03b00a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f03bc000c00 tx=0x7f03bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:20.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.188+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f03b809ba60 con 0x7f03c8102510 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/3516176185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 msgr2=0x7f0e1406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/3516176185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e1406ac30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009990 tx=0x7f0e00009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/3516176185 shutdown_connections 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/3516176185 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 0x7f0e1410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/3516176185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/3516176185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e1406ac30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/3516176185 >> 172.21.15.107:0/3516176185 conn(0x7f0e140780a0 msgr2=0x7f0e140784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/3516176185 shutdown_connections 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.137+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/3516176185 wait complete. 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 Processor -- start 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 -- start start 2024-04-24T23:44:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 0x7f0e14101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e141125a0 con 0x7f0e1406a7d0 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0e14112420 con 0x7f0e141082a0 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e1ad9b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0e14112720 con 0x7f0e1406b170 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e18b37700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e18b37700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45406/0 (socket says 172.21.15.107:45406) 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.138+0000 7f0e18b37700 1 -- 172.21.15.107:0/1396330634 learned_addr learned my addr 172.21.15.107:0/1396330634 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e19338700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e13fff700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 0x7f0e14101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 -- 172.21.15.107:0/1396330634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 msgr2=0x7f0e14101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 0x7f0e14101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 -- 172.21.15.107:0/1396330634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 msgr2=0x7f0e14101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 -- 172.21.15.107:0/1396330634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e141a6f20 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e19338700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e18b37700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800c9b0 tx=0x7f0e0800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08007b00 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e141a7210 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e08007ca0 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08018780 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.139+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e141a77a0 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.140+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e14068f20 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.142+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0e0800f450 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.142+0000 7f0e11ffb700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 0x7f0e040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.143+0000 7f0e13fff700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 0x7f0e040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.143+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0e080cf860 con 0x7f0e1406a7d0 2024-04-24T23:44:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.143+0000 7f0e13fff700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 0x7f0e040a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0e00006010 tx=0x7f0e00005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.148+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e08014030 con 0x7f0e1406a7d0 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.356+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0e1402db30 con 0x7f0e1406a7d0 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.357+0000 7f0e11ffb700 1 -- 172.21.15.107:0/1396330634 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0e0809c390 con 0x7f0e1406a7d0 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stdout:197568495673 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 msgr2=0x7f0e040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 0x7f0e040a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0e00006010 tx=0x7f0e00005e80 comp rx=0 tx=0).stop 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 msgr2=0x7f0e141011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800c9b0 tx=0x7f0e0800cd70 comp rx=0 tx=0).stop 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 shutdown_connections 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0e040a2cb0 0x7f0e040a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e141082a0 0x7f0e14101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e1406a7d0 0x7f0e141011f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 --2- 172.21.15.107:0/1396330634 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0e1406b170 0x7f0e14101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 >> 172.21.15.107:0/1396330634 conn(0x7f0e140780a0 msgr2=0x7f0e140ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 shutdown_connections 2024-04-24T23:44:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:18.359+0000 7f0e1ad9b700 1 -- 172.21.15.107:0/1396330634 wait complete. 2024-04-24T23:44:20.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.390+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f03c802db30 con 0x7f03c8102510 2024-04-24T23:44:20.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.390+0000 7f03c57fa700 1 -- 172.21.15.107:0/3082217731 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f03b809b360 con 0x7f03c8102510 2024-04-24T23:44:20.390 INFO:teuthology.orchestra.run.smithi107.stdout:55834574929 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 msgr2=0x7f03b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 0x7f03b00a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f03bc000c00 tx=0x7f03bc005fb0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 msgr2=0x7f03c819e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f03b800d950 tx=0x7f03b800dc60 comp rx=0 tx=0).stop 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 shutdown_connections 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f03b00a2cb0 0x7f03b00a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03c81075e0 0x7f03c8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c8102510 0x7f03c819e750 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 --2- 172.21.15.107:0/3082217731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f03c8102ec0 0x7f03c819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 >> 172.21.15.107:0/3082217731 conn(0x7f03c80793e0 msgr2=0x7f03c81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 shutdown_connections 2024-04-24T23:44:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:20.392+0000 7f03ce52c700 1 -- 172.21.15.107:0/3082217731 wait complete. 2024-04-24T23:44:20.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:20 smithi107 ceph-mon[29118]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:20.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:20 smithi195 ceph-mon[33019]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:20.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:20 smithi170 ceph-mon[32714]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:21.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:21 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3082217731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T23:44:21.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:21 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3082217731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T23:44:21.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:21 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3082217731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T23:44:22.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757309 for osd.5 2024-04-24T23:44:22.189 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:22.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:22 smithi107 ceph-mon[29118]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:22.757 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495673 for osd.6 2024-04-24T23:44:22.757 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:22.814 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:22 smithi195 ceph-mon[33019]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:22 smithi170 ceph-mon[32714]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:23.173 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574929 for osd.1 2024-04-24T23:44:23.173 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:23.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738454 for osd.0 2024-04-24T23:44:23.761 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 -- 172.21.15.107:0/59161965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 msgr2=0x7f8550107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 --2- 172.21.15.107:0/59161965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550107580 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8540009990 tx=0x7f8540009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 -- 172.21.15.107:0/59161965 shutdown_connections 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 --2- 172.21.15.107:0/59161965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 0x7f855010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 --2- 172.21.15.107:0/59161965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550107580 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 --2- 172.21.15.107:0/59161965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f85501015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.972+0000 7f855666e700 1 -- 172.21.15.107:0/59161965 >> 172.21.15.107:0/59161965 conn(0x7f85500780e0 msgr2=0x7f85500784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- 172.21.15.107:0/59161965 shutdown_connections 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- 172.21.15.107:0/59161965 wait complete. 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 Processor -- start 2024-04-24T23:44:23.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- start start 2024-04-24T23:44:23.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f8550102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:23.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 0x7f8550103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8550111440 con 0x7f85501071a0 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f85501112c0 con 0x7f8550101130 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.973+0000 7f855666e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f85501115c0 con 0x7f8550101af0 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45458/0 (socket says 172.21.15.107:45458) 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 -- 172.21.15.107:0/2248505085 learned_addr learned my addr 172.21.15.107:0/2248505085 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854ffff700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f8550102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854f7fe700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 0x7f8550103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 -- 172.21.15.107:0/2248505085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 msgr2=0x7f8550103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 0x7f8550103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 -- 172.21.15.107:0/2248505085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 msgr2=0x7f8550102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:23.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f8550102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 -- 172.21.15.107:0/2248505085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85501a6e50 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854ffff700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f8550102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f8554c0b700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f854800ada0 tx=0x7f854800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548017020 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8548004bb0 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85501a7140 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548005520 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.974+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85501a7680 con 0x7f85501071a0 2024-04-24T23:44:23.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.975+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8550068f20 con 0x7f85501071a0 2024-04-24T23:44:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.976+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f85480056c0 con 0x7f85501071a0 2024-04-24T23:44:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.977+0000 7f854d7fa700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 0x7f85380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.977+0000 7f854ffff700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 0x7f85380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.977+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f85480cf1f0 con 0x7f85501071a0 2024-04-24T23:44:23.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.977+0000 7f854ffff700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 0x7f85380a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8540006010 tx=0x7f854000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:23.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:23.980+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8548021020 con 0x7f85501071a0 2024-04-24T23:44:24.064 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:24.064 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:24.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.199+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f85501a79b0 con 0x7f85501071a0 2024-04-24T23:44:24.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.199+0000 7f854d7fa700 1 -- 172.21.15.107:0/2248505085 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f854809bd20 con 0x7f85501071a0 2024-04-24T23:44:24.199 INFO:teuthology.orchestra.run.smithi107.stdout:287762808875 2024-04-24T23:44:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 msgr2=0x7f85380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 0x7f85380a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8540006010 tx=0x7f854000b540 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 msgr2=0x7f8550103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f854800ada0 tx=0x7f854800c330 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 shutdown_connections 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f85380a2cb0 0x7f85380a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85501071a0 0x7f8550103890 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8550101130 0x7f8550102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 --2- 172.21.15.107:0/2248505085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8550101af0 0x7f8550103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 >> 172.21.15.107:0/2248505085 conn(0x7f85500780e0 msgr2=0x7f85501003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 shutdown_connections 2024-04-24T23:44:24.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:24.202+0000 7f855666e700 1 -- 172.21.15.107:0/2248505085 wait complete. 2024-04-24T23:44:25.012 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808875 for osd.9 2024-04-24T23:44:25.012 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:25 smithi195 ceph-mon[33019]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:25.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:25 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2248505085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T23:44:25.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:25 smithi107 ceph-mon[29118]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:25.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:25 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2248505085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T23:44:25.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:25 smithi170 ceph-mon[32714]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:25.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:25 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2248505085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T23:44:27.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:27 smithi195 ceph-mon[33019]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:27.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:27 smithi170 ceph-mon[32714]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:27.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:27 smithi107 ceph-mon[29118]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/997929719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101af0 msgr2=0x7f4a5410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/997929719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101af0 0x7f4a5410dfe0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00b0a0 tx=0x7f4a4c00b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/997929719 shutdown_connections 2024-04-24T23:44:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/997929719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101af0 0x7f4a5410dfe0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/997929719 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a541071a0 0x7f4a54107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/997929719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a54101130 0x7f4a541015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.133+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/997929719 >> 172.21.15.107:0/997929719 conn(0x7f4a540780e0 msgr2=0x7f4a540784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/997929719 shutdown_connections 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/997929719 wait complete. 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 Processor -- start 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- start start 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a54101af0 0x7f4a54199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a541071a0 0x7f4a541a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111510 con 0x7f4a54101130 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111390 con 0x7f4a541071a0 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5aaa1700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111690 con 0x7f4a54101af0 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5883d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5883d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60192/0 (socket says 172.21.15.107:60192) 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.134+0000 7f4a5883d700 1 -- 172.21.15.107:0/2881457764 learned_addr learned my addr 172.21.15.107:0/2881457764 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a53fff700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a54101af0 0x7f4a54199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 -- 172.21.15.107:0/2881457764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a54101af0 msgr2=0x7f4a54199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a54101af0 0x7f4a54199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 -- 172.21.15.107:0/2881457764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a541071a0 msgr2=0x7f4a541a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a541071a0 0x7f4a541a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 -- 172.21.15.107:0/2881457764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a541a4f50 con 0x7f4a54101130 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5883d700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800b530 tx=0x7f4a4800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48015020 con 0x7f4a54101130 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a48011d10 con 0x7f4a54101130 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a541a5120 con 0x7f4a54101130 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800f2e0 con 0x7f4a54101130 2024-04-24T23:44:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.135+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a541a55e0 con 0x7f4a54101130 2024-04-24T23:44:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.136+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54109710 con 0x7f4a54101130 2024-04-24T23:44:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.137+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4a48018020 con 0x7f4a54101130 2024-04-24T23:44:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.138+0000 7f4a51ffb700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 0x7f4a440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.138+0000 7f4a53fff700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 0x7f4a440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.138+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4a480ced10 con 0x7f4a54101130 2024-04-24T23:44:28.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.138+0000 7f4a53fff700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 0x7f4a440a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a54078570 tx=0x7f4a40009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:28.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.142+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a4801f020 con 0x7f4a54101130 2024-04-24T23:44:28.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.345+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4a5406a500 con 0x7f4a54101130 2024-04-24T23:44:28.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.345+0000 7f4a51ffb700 1 -- 172.21.15.107:0/2881457764 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4a4809b8c0 con 0x7f4a54101130 2024-04-24T23:44:28.345 INFO:teuthology.orchestra.run.smithi107.stdout:107374182473 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 msgr2=0x7f4a440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 0x7f4a440a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a54078570 tx=0x7f4a40009190 comp rx=0 tx=0).stop 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 msgr2=0x7f4a54199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800b530 tx=0x7f4a4800b840 comp rx=0 tx=0).stop 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 shutdown_connections 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4a440a2cb0 0x7f4a440a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a541071a0 0x7f4a541a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a54101130 0x7f4a54199990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 --2- 172.21.15.107:0/2881457764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4a54101af0 0x7f4a54199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 >> 172.21.15.107:0/2881457764 conn(0x7f4a540780e0 msgr2=0x7f4a540ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.348+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 shutdown_connections 2024-04-24T23:44:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.349+0000 7f4a5aaa1700 1 -- 172.21.15.107:0/2881457764 wait complete. 2024-04-24T23:44:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 -- 172.21.15.107:0/2195088890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0101ae0 msgr2=0x7f4bb010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/2195088890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0101ae0 0x7f4bb010dfd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800b0a0 tx=0x7f4ba800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 -- 172.21.15.107:0/2195088890 shutdown_connections 2024-04-24T23:44:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/2195088890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0101ae0 0x7f4bb010dfd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/2195088890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0107190 0x7f4bb0107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/2195088890 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101120 0x7f4bb01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 -- 172.21.15.107:0/2195088890 >> 172.21.15.107:0/2195088890 conn(0x7f4bb00780a0 msgr2=0x7f4bb00784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 -- 172.21.15.107:0/2195088890 shutdown_connections 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.354+0000 7f4bb6b28700 1 -- 172.21.15.107:0/2195088890 wait complete. 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 Processor -- start 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 -- start start 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 0x7f4bb019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101ae0 0x7f4bb019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb0111430 con 0x7f4bb0107190 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb01112b0 con 0x7f4bb0101120 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb6b28700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb01115b0 con 0x7f4bb0101ae0 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60206/0 (socket says 172.21.15.107:60206) 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 -- 172.21.15.107:0/948234113 learned_addr learned my addr 172.21.15.107:0/948234113 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb48c4700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 0x7f4bb019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4baffff700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101ae0 0x7f4bb019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 -- 172.21.15.107:0/948234113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101ae0 msgr2=0x7f4bb019db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101ae0 0x7f4bb019db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 -- 172.21.15.107:0/948234113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 msgr2=0x7f4bb019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 0x7f4bb019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb50c5700 1 -- 172.21.15.107:0/948234113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb0197f70 con 0x7f4bb0107190 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.355+0000 7f4bb48c4700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 0x7f4bb019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4bb50c5700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800b780 tx=0x7f4ba8004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8021070 con 0x7f4bb0107190 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb0198200 con 0x7f4bb0107190 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba8007c80 con 0x7f4bb0107190 2024-04-24T23:44:28.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8005400 con 0x7f4bb0107190 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.356+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb01a4540 con 0x7f4bb0107190 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.357+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb0102840 con 0x7f4bb0107190 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.358+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4ba8024070 con 0x7f4bb0107190 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.359+0000 7f4badffb700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 0x7f4ba00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.359+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4ba800f020 con 0x7f4bb0107190 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.359+0000 7f4bb48c4700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 0x7f4ba00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.359+0000 7f4bb48c4700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 0x7f4ba00a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005e10 tx=0x7f4ba4005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:28.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.362+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ba802b020 con 0x7f4bb0107190 2024-04-24T23:44:28.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.565+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f4bb0198bc0 con 0x7f4bb0107190 2024-04-24T23:44:28.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.565+0000 7f4badffb700 1 -- 172.21.15.107:0/948234113 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4ba809da50 con 0x7f4bb0107190 2024-04-24T23:44:28.565 INFO:teuthology.orchestra.run.smithi107.stdout:343597383712 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 msgr2=0x7f4ba00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 0x7f4ba00a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005e10 tx=0x7f4ba4005d60 comp rx=0 tx=0).stop 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 msgr2=0x7f4bb0197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800b780 tx=0x7f4ba8004db0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 shutdown_connections 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba00a2cb0 0x7f4ba00a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb0107190 0x7f4bb0197690 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bb0101120 0x7f4bb019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 --2- 172.21.15.107:0/948234113 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bb0101ae0 0x7f4bb019db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.567+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 >> 172.21.15.107:0/948234113 conn(0x7f4bb00780a0 msgr2=0x7f4bb00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.568+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 shutdown_connections 2024-04-24T23:44:28.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:28.568+0000 7f4bb6b28700 1 -- 172.21.15.107:0/948234113 wait complete. 2024-04-24T23:44:29.247 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:29 smithi107 ceph-mon[29118]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:29.247 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:29 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2881457764' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T23:44:29.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:29 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/948234113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T23:44:29.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383708 got 343597383712 for osd.11 2024-04-24T23:44:29.276 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:29 smithi195 ceph-mon[33019]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:29 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2881457764' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T23:44:29.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:29 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/948234113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T23:44:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:29 smithi170 ceph-mon[32714]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:29 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2881457764' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T23:44:29.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:29 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/948234113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T23:44:29.755 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182473 for osd.3 2024-04-24T23:44:29.756 DEBUG:teuthology.parallel:result is None 2024-04-24T23:44:29.756 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-24T23:44:29.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T23:44:29.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:31.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:31 smithi107 ceph-mon[29118]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:31.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:31 smithi195 ceph-mon[33019]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:31.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:31 smithi170 ceph-mon[32714]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-24T23:44:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- 172.21.15.107:0/1440274697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 msgr2=0x7fcdb8106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/1440274697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8106e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009990 tx=0x7fcdac009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- 172.21.15.107:0/1440274697 shutdown_connections 2024-04-24T23:44:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/1440274697 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 0x7fcdb810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/1440274697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 0x7fcdb81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/1440274697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8106e90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- 172.21.15.107:0/1440274697 >> 172.21.15.107:0/1440274697 conn(0x7fcdb8079390 msgr2=0x7fcdb80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- 172.21.15.107:0/1440274697 shutdown_connections 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- 172.21.15.107:0/1440274697 wait complete. 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 Processor -- start 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.714+0000 7fcdbec0b700 1 -- start start 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 0x7fcdb8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 0x7fcdb8074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8112590 con 0x7fcdb8102e90 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8112410 con 0x7fcdb81024e0 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb8112710 con 0x7fcdb8107570 2024-04-24T23:44:31.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60226/0 (socket says 172.21.15.107:60226) 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 -- 172.21.15.107:0/2596786387 learned_addr learned my addr 172.21.15.107:0/2596786387 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbc9a7700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 0x7fcdb8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbd1a8700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 0x7fcdb8074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 -- 172.21.15.107:0/2596786387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 msgr2=0x7fcdb8074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 0x7fcdb8074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 -- 172.21.15.107:0/2596786387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 msgr2=0x7fcdb8074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 0x7fcdb8074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 -- 172.21.15.107:0/2596786387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb81a6fb0 con 0x7fcdb8102e90 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb7fff700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcdac000c00 tx=0x7fcdac003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac027070 con 0x7fcdb8102e90 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb81a7240 con 0x7fcdb8102e90 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdac004050 con 0x7fcdb8102e90 2024-04-24T23:44:31.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.715+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac00f460 con 0x7fcdb8102e90 2024-04-24T23:44:31.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.716+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a7690 con 0x7fcdb8102e90 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.717+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb8103970 con 0x7fcdb8102e90 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.718+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fcdac02a070 con 0x7fcdb8102e90 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.719+0000 7fcdb5ffb700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 0x7fcda00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.719+0000 7fcdbc9a7700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 0x7fcda00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.719+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcdac0d0950 con 0x7fcdb8102e90 2024-04-24T23:44:31.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.720+0000 7fcdbc9a7700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 0x7fcda00a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009a10 tx=0x7fcda8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:31.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.723+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdac031070 con 0x7fcdb8102e90 2024-04-24T23:44:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.916+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcdb8062670 con 0x7fcda00a2cb0 2024-04-24T23:44:31.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.926+0000 7fcdb5ffb700 1 -- 172.21.15.107:0/2596786387 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340702 (secure 0 0 0) 0x7fcdb8062670 con 0x7fcda00a2cb0 2024-04-24T23:44:31.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 msgr2=0x7fcda00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 0x7fcda00a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009a10 tx=0x7fcda8009190 comp rx=0 tx=0).stop 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 msgr2=0x7fcdb8074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcdac000c00 tx=0x7fcdac003680 comp rx=0 tx=0).stop 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 shutdown_connections 2024-04-24T23:44:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcda00a2cb0 0x7fcda00a5170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcdb8107570 0x7fcdb8074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb81024e0 0x7fcdb8074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 --2- 172.21.15.107:0/2596786387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb8102e90 0x7fcdb8074a70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 >> 172.21.15.107:0/2596786387 conn(0x7fcdb8079390 msgr2=0x7fcdb8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 shutdown_connections 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:31.929+0000 7fcdbec0b700 1 -- 172.21.15.107:0/2596786387 wait complete. 2024-04-24T23:44:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:44:32.773 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-24T23:44:29.999576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659748,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":848,"num_read_kb":540,"num_write":424,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1180848,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365680,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576144,"statfs":{"total":1151940427776,"available":1151565971456,"internally_reserved":0,"allocated":12591104,"data_stored":7965050,"data_compressed":361080,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":54,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223282+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:43:49.223282+0000","last_peered":"2024-04-24T23:43:49.223282+0000","last_clean":"2024-04-24T23:43:49.223282+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:43:49.223282+0000","last_undegraded":"2024-04-24T23:43:49.223282+0000","last_fullsized":"2024-04-24T23:43:49.223282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228886+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:43:49.228886+0000","last_peered":"2024-04-24T23:43:49.228886+0000","last_clean":"2024-04-24T23:43:49.228886+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:43:49.228886+0000","last_undegraded":"2024-04-24T23:43:49.228886+0000","last_fullsized":"2024-04-24T23:43:49.228886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228134+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:43:49.228134+0000","last_peered":"2024-04-24T23:43:49.228134+0000","last_clean":"2024-04-24T23:43:49.228134+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:43:49.228134+0000","last_undegraded":"2024-04-24T23:43:49.228134+0000","last_fullsized":"2024-04-24T23:43:49.228134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217676+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:43:49.217676+0000","last_peered":"2024-04-24T23:43:49.217676+0000","last_clean":"2024-04-24T23:43:49.217676+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:43:49.217676+0000","last_undegraded":"2024-04-24T23:43:49.217676+0000","last_fullsized":"2024-04-24T23:43:49.217676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219695+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:43:49.219695+0000","last_peered":"2024-04-24T23:43:49.219695+0000","last_clean":"2024-04-24T23:43:49.219695+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:43:49.219695+0000","last_undegraded":"2024-04-24T23:43:49.219695+0000","last_fullsized":"2024-04-24T23:43:49.219695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274694+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:43:49.274694+0000","last_peered":"2024-04-24T23:43:49.274694+0000","last_clean":"2024-04-24T23:43:49.274694+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:43:49.274694+0000","last_undegraded":"2024-04-24T23:43:49.274694+0000","last_fullsized":"2024-04-24T23:43:49.274694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218872+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:43:49.218872+0000","last_peered":"2024-04-24T23:43:49.218872+0000","last_clean":"2024-04-24T23:43:49.218872+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:43:49.218872+0000","last_undegraded":"2024-04-24T23:43:49.218872+0000","last_fullsized":"2024-04-24T23:43:49.218872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224087+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:43:49.224087+0000","last_peered":"2024-04-24T23:43:49.224087+0000","last_clean":"2024-04-24T23:43:49.224087+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:43:49.224087+0000","last_undegraded":"2024-04-24T23:43:49.224087+0000","last_fullsized":"2024-04-24T23:43:49.224087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228226+0000","last_change":"2024-04-24T23:43:49.227407+0000","last_active":"2024-04-24T23:43:49.228226+0000","last_peered":"2024-04-24T23:43:49.228226+0000","last_clean":"2024-04-24T23:43:49.228226+0000","last_became_active":"2024-04-24T23:43:49.227076+0000","last_became_peered":"2024-04-24T23:43:49.227076+0000","last_unstale":"2024-04-24T23:43:49.228226+0000","last_undegraded":"2024-04-24T23:43:49.228226+0000","last_fullsized":"2024-04-24T23:43:49.228226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:24.426521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.275170+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:43:49.275170+0000","last_peered":"2024-04-24T23:43:49.275170+0000","last_clean":"2024-04-24T23:43:49.275170+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:43:49.275170+0000","last_undegraded":"2024-04-24T23:43:49.275170+0000","last_fullsized":"2024-04-24T23:43:49.275170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219308+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:43:49.219308+0000","last_peered":"2024-04-24T23:43:49.219308+0000","last_clean":"2024-04-24T23:43:49.219308+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:43:49.219308+0000","last_undegraded":"2024-04-24T23:43:49.219308+0000","last_fullsized":"2024-04-24T23:43:49.219308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217511+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:43:49.217511+0000","last_peered":"2024-04-24T23:43:49.217511+0000","last_clean":"2024-04-24T23:43:49.217511+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:43:49.217511+0000","last_undegraded":"2024-04-24T23:43:49.217511+0000","last_fullsized":"2024-04-24T23:43:49.217511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220072+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:43:49.220072+0000","last_peered":"2024-04-24T23:43:49.220072+0000","last_clean":"2024-04-24T23:43:49.220072+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:43:49.220072+0000","last_undegraded":"2024-04-24T23:43:49.220072+0000","last_fullsized":"2024-04-24T23:43:49.220072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228190+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:43:49.228190+0000","last_peered":"2024-04-24T23:43:49.228190+0000","last_clean":"2024-04-24T23:43:49.228190+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:43:49.228190+0000","last_undegraded":"2024-04-24T23:43:49.228190+0000","last_fullsized":"2024-04-24T23:43:49.228190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228022+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:43:49.228022+0000","last_peered":"2024-04-24T23:43:49.228022+0000","last_clean":"2024-04-24T23:43:49.228022+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:43:49.228022+0000","last_undegraded":"2024-04-24T23:43:49.228022+0000","last_fullsized":"2024-04-24T23:43:49.228022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221339+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:43:49.221339+0000","last_peered":"2024-04-24T23:43:49.221339+0000","last_clean":"2024-04-24T23:43:49.221339+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:43:49.221339+0000","last_undegraded":"2024-04-24T23:43:49.221339+0000","last_fullsized":"2024-04-24T23:43:49.221339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223180+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:43:49.223180+0000","last_peered":"2024-04-24T23:43:49.223180+0000","last_clean":"2024-04-24T23:43:49.223180+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:43:49.223180+0000","last_undegraded":"2024-04-24T23:43:49.223180+0000","last_fullsized":"2024-04-24T23:43:49.223180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220249+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:43:49.220249+0000","last_peered":"2024-04-24T23:43:49.220249+0000","last_clean":"2024-04-24T23:43:49.220249+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:43:49.220249+0000","last_undegraded":"2024-04-24T23:43:49.220249+0000","last_fullsized":"2024-04-24T23:43:49.220249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217448+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:43:49.217448+0000","last_peered":"2024-04-24T23:43:49.217448+0000","last_clean":"2024-04-24T23:43:49.217448+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:43:49.217448+0000","last_undegraded":"2024-04-24T23:43:49.217448+0000","last_fullsized":"2024-04-24T23:43:49.217448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216955+0000","last_change":"2024-04-24T23:42:55.327166+0000","last_active":"2024-04-24T23:43:49.216955+0000","last_peered":"2024-04-24T23:43:49.216955+0000","last_clean":"2024-04-24T23:43:49.216955+0000","last_became_active":"2024-04-24T23:42:55.326708+0000","last_became_peered":"2024-04-24T23:42:55.326708+0000","last_unstale":"2024-04-24T23:43:49.216955+0000","last_undegraded":"2024-04-24T23:43:49.216955+0000","last_fullsized":"2024-04-24T23:43:49.216955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:39.604161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:24.187532+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:44:24.187532+0000","last_peered":"2024-04-24T23:44:24.187532+0000","last_clean":"2024-04-24T23:44:24.187532+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:44:24.187532+0000","last_undegraded":"2024-04-24T23:44:24.187532+0000","last_fullsized":"2024-04-24T23:44:24.187532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224207+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:43:49.224207+0000","last_peered":"2024-04-24T23:43:49.224207+0000","last_clean":"2024-04-24T23:43:49.224207+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:43:49.224207+0000","last_undegraded":"2024-04-24T23:43:49.224207+0000","last_fullsized":"2024-04-24T23:43:49.224207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217033+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:43:49.217033+0000","last_peered":"2024-04-24T23:43:49.217033+0000","last_clean":"2024-04-24T23:43:49.217033+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:43:49.217033+0000","last_undegraded":"2024-04-24T23:43:49.217033+0000","last_fullsized":"2024-04-24T23:43:49.217033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224563+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:43:49.224563+0000","last_peered":"2024-04-24T23:43:49.224563+0000","last_clean":"2024-04-24T23:43:49.224563+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:43:49.224563+0000","last_undegraded":"2024-04-24T23:43:49.224563+0000","last_fullsized":"2024-04-24T23:43:49.224563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228890+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:43:49.228890+0000","last_peered":"2024-04-24T23:43:49.228890+0000","last_clean":"2024-04-24T23:43:49.228890+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:43:49.228890+0000","last_undegraded":"2024-04-24T23:43:49.228890+0000","last_fullsized":"2024-04-24T23:43:49.228890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217145+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:43:49.217145+0000","last_peered":"2024-04-24T23:43:49.217145+0000","last_clean":"2024-04-24T23:43:49.217145+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:43:49.217145+0000","last_undegraded":"2024-04-24T23:43:49.217145+0000","last_fullsized":"2024-04-24T23:43:49.217145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.235090+0000","last_change":"2024-04-24T23:43:49.235090+0000","last_active":"2024-04-24T23:43:49.235090+0000","last_peered":"2024-04-24T23:43:49.235090+0000","last_clean":"2024-04-24T23:43:49.235090+0000","last_became_active":"2024-04-24T23:43:49.223773+0000","last_became_peered":"2024-04-24T23:43:49.223773+0000","last_unstale":"2024-04-24T23:43:49.235090+0000","last_undegraded":"2024-04-24T23:43:49.235090+0000","last_fullsized":"2024-04-24T23:43:49.235090+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:23.776176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221725+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:43:49.221725+0000","last_peered":"2024-04-24T23:43:49.221725+0000","last_clean":"2024-04-24T23:43:49.221725+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:43:49.221725+0000","last_undegraded":"2024-04-24T23:43:49.221725+0000","last_fullsized":"2024-04-24T23:43:49.221725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217650+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:43:49.217650+0000","last_peered":"2024-04-24T23:43:49.217650+0000","last_clean":"2024-04-24T23:43:49.217650+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:43:49.217650+0000","last_undegraded":"2024-04-24T23:43:49.217650+0000","last_fullsized":"2024-04-24T23:43:49.217650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219975+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:43:49.219975+0000","last_peered":"2024-04-24T23:43:49.219975+0000","last_clean":"2024-04-24T23:43:49.219975+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:43:49.219975+0000","last_undegraded":"2024-04-24T23:43:49.219975+0000","last_fullsized":"2024-04-24T23:43:49.219975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220338+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:43:49.220338+0000","last_peered":"2024-04-24T23:43:49.220338+0000","last_clean":"2024-04-24T23:43:49.220338+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:43:49.220338+0000","last_undegraded":"2024-04-24T23:43:49.220338+0000","last_fullsized":"2024-04-24T23:43:49.220338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216846+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:43:49.216846+0000","last_peered":"2024-04-24T23:43:49.216846+0000","last_clean":"2024-04-24T23:43:49.216846+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:43:49.216846+0000","last_undegraded":"2024-04-24T23:43:49.216846+0000","last_fullsized":"2024-04-24T23:43:49.216846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:26.191650+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:44:26.191650+0000","last_peered":"2024-04-24T23:44:26.191650+0000","last_clean":"2024-04-24T23:44:26.191650+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:44:26.191650+0000","last_undegraded":"2024-04-24T23:44:26.191650+0000","last_fullsized":"2024-04-24T23:44:26.191650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224709+0000","last_change":"2024-04-24T23:43:49.224709+0000","last_active":"2024-04-24T23:43:49.224709+0000","last_peered":"2024-04-24T23:43:49.224709+0000","last_clean":"2024-04-24T23:43:49.224709+0000","last_became_active":"2024-04-24T23:43:49.224345+0000","last_became_peered":"2024-04-24T23:43:49.224345+0000","last_unstale":"2024-04-24T23:43:49.224709+0000","last_undegraded":"2024-04-24T23:43:49.224709+0000","last_fullsized":"2024-04-24T23:43:49.224709+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:56.093532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216961+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:43:49.216961+0000","last_peered":"2024-04-24T23:43:49.216961+0000","last_clean":"2024-04-24T23:43:49.216961+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:43:49.216961+0000","last_undegraded":"2024-04-24T23:43:49.216961+0000","last_fullsized":"2024-04-24T23:43:49.216961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274646+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:43:49.274646+0000","last_peered":"2024-04-24T23:43:49.274646+0000","last_clean":"2024-04-24T23:43:49.274646+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:43:49.274646+0000","last_undegraded":"2024-04-24T23:43:49.274646+0000","last_fullsized":"2024-04-24T23:43:49.274646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219945+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:43:49.219945+0000","last_peered":"2024-04-24T23:43:49.219945+0000","last_clean":"2024-04-24T23:43:49.219945+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:43:49.219945+0000","last_undegraded":"2024-04-24T23:43:49.219945+0000","last_fullsized":"2024-04-24T23:43:49.219945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223167+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:43:49.223167+0000","last_peered":"2024-04-24T23:43:49.223167+0000","last_clean":"2024-04-24T23:43:49.223167+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:43:49.223167+0000","last_undegraded":"2024-04-24T23:43:49.223167+0000","last_fullsized":"2024-04-24T23:43:49.223167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220307+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:43:49.220307+0000","last_peered":"2024-04-24T23:43:49.220307+0000","last_clean":"2024-04-24T23:43:49.220307+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:43:49.220307+0000","last_undegraded":"2024-04-24T23:43:49.220307+0000","last_fullsized":"2024-04-24T23:43:49.220307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.227375+0000","last_change":"2024-04-24T23:43:49.226164+0000","last_active":"2024-04-24T23:43:49.227375+0000","last_peered":"2024-04-24T23:43:49.227375+0000","last_clean":"2024-04-24T23:43:49.227375+0000","last_became_active":"2024-04-24T23:43:49.225806+0000","last_became_peered":"2024-04-24T23:43:49.225806+0000","last_unstale":"2024-04-24T23:43:49.227375+0000","last_undegraded":"2024-04-24T23:43:49.227375+0000","last_fullsized":"2024-04-24T23:43:49.227375+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:56.068947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228954+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:43:49.228954+0000","last_peered":"2024-04-24T23:43:49.228954+0000","last_clean":"2024-04-24T23:43:49.228954+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:43:49.228954+0000","last_undegraded":"2024-04-24T23:43:49.228954+0000","last_fullsized":"2024-04-24T23:43:49.228954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220952+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:43:49.220952+0000","last_peered":"2024-04-24T23:43:49.220952+0000","last_clean":"2024-04-24T23:43:49.220952+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:43:49.220952+0000","last_undegraded":"2024-04-24T23:43:49.220952+0000","last_fullsized":"2024-04-24T23:43:49.220952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219413+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:43:49.219413+0000","last_peered":"2024-04-24T23:43:49.219413+0000","last_clean":"2024-04-24T23:43:49.219413+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:43:49.219413+0000","last_undegraded":"2024-04-24T23:43:49.219413+0000","last_fullsized":"2024-04-24T23:43:49.219413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223595+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:43:49.223595+0000","last_peered":"2024-04-24T23:43:49.223595+0000","last_clean":"2024-04-24T23:43:49.223595+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:43:49.223595+0000","last_undegraded":"2024-04-24T23:43:49.223595+0000","last_fullsized":"2024-04-24T23:43:49.223595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":164,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:25.189637+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:44:25.189637+0000","last_peered":"2024-04-24T23:44:25.189637+0000","last_clean":"2024-04-24T23:44:25.189637+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:44:25.189637+0000","last_undegraded":"2024-04-24T23:44:25.189637+0000","last_fullsized":"2024-04-24T23:44:25.189637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":134,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221890+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:43:49.221890+0000","last_peered":"2024-04-24T23:43:49.221890+0000","last_clean":"2024-04-24T23:43:49.221890+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:43:49.221890+0000","last_undegraded":"2024-04-24T23:43:49.221890+0000","last_fullsized":"2024-04-24T23:43:49.221890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219304+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:43:49.219304+0000","last_peered":"2024-04-24T23:43:49.219304+0000","last_clean":"2024-04-24T23:43:49.219304+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:43:49.219304+0000","last_undegraded":"2024-04-24T23:43:49.219304+0000","last_fullsized":"2024-04-24T23:43:49.219304+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.226781+0000","last_change":"2024-04-24T23:43:49.226781+0000","last_active":"2024-04-24T23:43:49.226781+0000","last_peered":"2024-04-24T23:43:49.226781+0000","last_clean":"2024-04-24T23:43:49.226781+0000","last_became_active":"2024-04-24T23:43:49.222089+0000","last_became_peered":"2024-04-24T23:43:49.222089+0000","last_unstale":"2024-04-24T23:43:49.226781+0000","last_undegraded":"2024-04-24T23:43:49.226781+0000","last_fullsized":"2024-04-24T23:43:49.226781+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:57.224621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228886+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:43:49.228886+0000","last_peered":"2024-04-24T23:43:49.228886+0000","last_clean":"2024-04-24T23:43:49.228886+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:43:49.228886+0000","last_undegraded":"2024-04-24T23:43:49.228886+0000","last_fullsized":"2024-04-24T23:43:49.228886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217287+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:43:49.217287+0000","last_peered":"2024-04-24T23:43:49.217287+0000","last_clean":"2024-04-24T23:43:49.217287+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:43:49.217287+0000","last_undegraded":"2024-04-24T23:43:49.217287+0000","last_fullsized":"2024-04-24T23:43:49.217287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220237+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:43:49.220237+0000","last_peered":"2024-04-24T23:43:49.220237+0000","last_clean":"2024-04-24T23:43:49.220237+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:43:49.220237+0000","last_undegraded":"2024-04-24T23:43:49.220237+0000","last_fullsized":"2024-04-24T23:43:49.220237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221866+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:43:49.221866+0000","last_peered":"2024-04-24T23:43:49.221866+0000","last_clean":"2024-04-24T23:43:49.221866+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:43:49.221866+0000","last_undegraded":"2024-04-24T23:43:49.221866+0000","last_fullsized":"2024-04-24T23:43:49.221866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219577+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:43:49.219577+0000","last_peered":"2024-04-24T23:43:49.219577+0000","last_clean":"2024-04-24T23:43:49.219577+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:43:49.219577+0000","last_undegraded":"2024-04-24T23:43:49.219577+0000","last_fullsized":"2024-04-24T23:43:49.219577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":202,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:25.189735+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:44:25.189735+0000","last_peered":"2024-04-24T23:44:25.189735+0000","last_clean":"2024-04-24T23:44:25.189735+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:44:25.189735+0000","last_undegraded":"2024-04-24T23:44:25.189735+0000","last_fullsized":"2024-04-24T23:44:25.189735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":141,"num_read_kb":131,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221537+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:43:49.221537+0000","last_peered":"2024-04-24T23:43:49.221537+0000","last_clean":"2024-04-24T23:43:49.221537+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:43:49.221537+0000","last_undegraded":"2024-04-24T23:43:49.221537+0000","last_fullsized":"2024-04-24T23:43:49.221537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220665+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:43:49.220665+0000","last_peered":"2024-04-24T23:43:49.220665+0000","last_clean":"2024-04-24T23:43:49.220665+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:43:49.220665+0000","last_undegraded":"2024-04-24T23:43:49.220665+0000","last_fullsized":"2024-04-24T23:43:49.220665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216901+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:43:49.216901+0000","last_peered":"2024-04-24T23:43:49.216901+0000","last_clean":"2024-04-24T23:43:49.216901+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:43:49.216901+0000","last_undegraded":"2024-04-24T23:43:49.216901+0000","last_fullsized":"2024-04-24T23:43:49.216901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219612+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:43:49.219612+0000","last_peered":"2024-04-24T23:43:49.219612+0000","last_clean":"2024-04-24T23:43:49.219612+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:43:49.219612+0000","last_undegraded":"2024-04-24T23:43:49.219612+0000","last_fullsized":"2024-04-24T23:43:49.219612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223064+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:43:49.223064+0000","last_peered":"2024-04-24T23:43:49.223064+0000","last_clean":"2024-04-24T23:43:49.223064+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:43:49.223064+0000","last_undegraded":"2024-04-24T23:43:49.223064+0000","last_fullsized":"2024-04-24T23:43:49.223064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219484+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:43:49.219484+0000","last_peered":"2024-04-24T23:43:49.219484+0000","last_clean":"2024-04-24T23:43:49.219484+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:43:49.219484+0000","last_undegraded":"2024-04-24T23:43:49.219484+0000","last_fullsized":"2024-04-24T23:43:49.219484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219229+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:43:49.219229+0000","last_peered":"2024-04-24T23:43:49.219229+0000","last_clean":"2024-04-24T23:43:49.219229+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:43:49.219229+0000","last_undegraded":"2024-04-24T23:43:49.219229+0000","last_fullsized":"2024-04-24T23:43:49.219229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220787+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:43:49.220787+0000","last_peered":"2024-04-24T23:43:49.220787+0000","last_clean":"2024-04-24T23:43:49.220787+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:43:49.220787+0000","last_undegraded":"2024-04-24T23:43:49.220787+0000","last_fullsized":"2024-04-24T23:43:49.220787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:27.192864+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:44:27.192864+0000","last_peered":"2024-04-24T23:44:27.192864+0000","last_clean":"2024-04-24T23:44:27.192864+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:44:27.192864+0000","last_undegraded":"2024-04-24T23:44:27.192864+0000","last_fullsized":"2024-04-24T23:44:27.192864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218968+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:43:49.218968+0000","last_peered":"2024-04-24T23:43:49.218968+0000","last_clean":"2024-04-24T23:43:49.218968+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:43:49.218968+0000","last_undegraded":"2024-04-24T23:43:49.218968+0000","last_fullsized":"2024-04-24T23:43:49.218968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220615+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:43:49.220615+0000","last_peered":"2024-04-24T23:43:49.220615+0000","last_clean":"2024-04-24T23:43:49.220615+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:43:49.220615+0000","last_undegraded":"2024-04-24T23:43:49.220615+0000","last_fullsized":"2024-04-24T23:43:49.220615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219577+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:43:49.219577+0000","last_peered":"2024-04-24T23:43:49.219577+0000","last_clean":"2024-04-24T23:43:49.219577+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:43:49.219577+0000","last_undegraded":"2024-04-24T23:43:49.219577+0000","last_fullsized":"2024-04-24T23:43:49.219577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223137+0000","last_change":"2024-04-24T23:43:49.223137+0000","last_active":"2024-04-24T23:43:49.223137+0000","last_peered":"2024-04-24T23:43:49.223137+0000","last_clean":"2024-04-24T23:43:49.223137+0000","last_became_active":"2024-04-24T23:43:49.222775+0000","last_became_peered":"2024-04-24T23:43:49.222775+0000","last_unstale":"2024-04-24T23:43:49.223137+0000","last_undegraded":"2024-04-24T23:43:49.223137+0000","last_fullsized":"2024-04-24T23:43:49.223137+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:22.910320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219413+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:43:49.219413+0000","last_peered":"2024-04-24T23:43:49.219413+0000","last_clean":"2024-04-24T23:43:49.219413+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:43:49.219413+0000","last_undegraded":"2024-04-24T23:43:49.219413+0000","last_fullsized":"2024-04-24T23:43:49.219413+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221934+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:43:49.221934+0000","last_peered":"2024-04-24T23:43:49.221934+0000","last_clean":"2024-04-24T23:43:49.221934+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:43:49.221934+0000","last_undegraded":"2024-04-24T23:43:49.221934+0000","last_fullsized":"2024-04-24T23:43:49.221934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224309+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:43:49.224309+0000","last_peered":"2024-04-24T23:43:49.224309+0000","last_clean":"2024-04-24T23:43:49.224309+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:43:49.224309+0000","last_undegraded":"2024-04-24T23:43:49.224309+0000","last_fullsized":"2024-04-24T23:43:49.224309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219939+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:43:49.219939+0000","last_peered":"2024-04-24T23:43:49.219939+0000","last_clean":"2024-04-24T23:43:49.219939+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:43:49.219939+0000","last_undegraded":"2024-04-24T23:43:49.219939+0000","last_fullsized":"2024-04-24T23:43:49.219939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220984+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:43:49.220984+0000","last_peered":"2024-04-24T23:43:49.220984+0000","last_clean":"2024-04-24T23:43:49.220984+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:43:49.220984+0000","last_undegraded":"2024-04-24T23:43:49.220984+0000","last_fullsized":"2024-04-24T23:43:49.220984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274963+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:43:49.274963+0000","last_peered":"2024-04-24T23:43:49.274963+0000","last_clean":"2024-04-24T23:43:49.274963+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:43:49.274963+0000","last_undegraded":"2024-04-24T23:43:49.274963+0000","last_fullsized":"2024-04-24T23:43:49.274963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220029+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:43:49.220029+0000","last_peered":"2024-04-24T23:43:49.220029+0000","last_clean":"2024-04-24T23:43:49.220029+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:43:49.220029+0000","last_undegraded":"2024-04-24T23:43:49.220029+0000","last_fullsized":"2024-04-24T23:43:49.220029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:28.193538+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:44:28.193538+0000","last_peered":"2024-04-24T23:44:28.193538+0000","last_clean":"2024-04-24T23:44:28.193538+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:44:28.193538+0000","last_undegraded":"2024-04-24T23:44:28.193538+0000","last_fullsized":"2024-04-24T23:44:28.193538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274749+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:43:49.274749+0000","last_peered":"2024-04-24T23:43:49.274749+0000","last_clean":"2024-04-24T23:43:49.274749+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:43:49.274749+0000","last_undegraded":"2024-04-24T23:43:49.274749+0000","last_fullsized":"2024-04-24T23:43:49.274749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220729+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:43:49.220729+0000","last_peered":"2024-04-24T23:43:49.220729+0000","last_clean":"2024-04-24T23:43:49.220729+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:43:49.220729+0000","last_undegraded":"2024-04-24T23:43:49.220729+0000","last_fullsized":"2024-04-24T23:43:49.220729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221044+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:43:49.221044+0000","last_peered":"2024-04-24T23:43:49.221044+0000","last_clean":"2024-04-24T23:43:49.221044+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:43:49.221044+0000","last_undegraded":"2024-04-24T23:43:49.221044+0000","last_fullsized":"2024-04-24T23:43:49.221044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:27.192365+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:44:27.192365+0000","last_peered":"2024-04-24T23:44:27.192365+0000","last_clean":"2024-04-24T23:44:27.192365+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:44:27.192365+0000","last_undegraded":"2024-04-24T23:44:27.192365+0000","last_fullsized":"2024-04-24T23:44:27.192365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219192+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:43:49.219192+0000","last_peered":"2024-04-24T23:43:49.219192+0000","last_clean":"2024-04-24T23:43:49.219192+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:43:49.219192+0000","last_undegraded":"2024-04-24T23:43:49.219192+0000","last_fullsized":"2024-04-24T23:43:49.219192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223644+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:43:49.223644+0000","last_peered":"2024-04-24T23:43:49.223644+0000","last_clean":"2024-04-24T23:43:49.223644+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:43:49.223644+0000","last_undegraded":"2024-04-24T23:43:49.223644+0000","last_fullsized":"2024-04-24T23:43:49.223644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217258+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:43:49.217258+0000","last_peered":"2024-04-24T23:43:49.217258+0000","last_clean":"2024-04-24T23:43:49.217258+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:43:49.217258+0000","last_undegraded":"2024-04-24T23:43:49.217258+0000","last_fullsized":"2024-04-24T23:43:49.217258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218904+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:43:49.218904+0000","last_peered":"2024-04-24T23:43:49.218904+0000","last_clean":"2024-04-24T23:43:49.218904+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:43:49.218904+0000","last_undegraded":"2024-04-24T23:43:49.218904+0000","last_fullsized":"2024-04-24T23:43:49.218904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220890+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:43:49.220890+0000","last_peered":"2024-04-24T23:43:49.220890+0000","last_clean":"2024-04-24T23:43:49.220890+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:43:49.220890+0000","last_undegraded":"2024-04-24T23:43:49.220890+0000","last_fullsized":"2024-04-24T23:43:49.220890+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219005+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:43:49.219005+0000","last_peered":"2024-04-24T23:43:49.219005+0000","last_clean":"2024-04-24T23:43:49.219005+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:43:49.219005+0000","last_undegraded":"2024-04-24T23:43:49.219005+0000","last_fullsized":"2024-04-24T23:43:49.219005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218892+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:43:49.218892+0000","last_peered":"2024-04-24T23:43:49.218892+0000","last_clean":"2024-04-24T23:43:49.218892+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:43:49.218892+0000","last_undegraded":"2024-04-24T23:43:49.218892+0000","last_fullsized":"2024-04-24T23:43:49.218892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.222949+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:43:49.222949+0000","last_peered":"2024-04-24T23:43:49.222949+0000","last_clean":"2024-04-24T23:43:49.222949+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:43:49.222949+0000","last_undegraded":"2024-04-24T23:43:49.222949+0000","last_fullsized":"2024-04-24T23:43:49.222949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.235755+0000","last_change":"2024-04-24T23:43:49.235755+0000","last_active":"2024-04-24T23:43:49.235755+0000","last_peered":"2024-04-24T23:43:49.235755+0000","last_clean":"2024-04-24T23:43:49.235755+0000","last_became_active":"2024-04-24T23:43:49.230488+0000","last_became_peered":"2024-04-24T23:43:49.230488+0000","last_unstale":"2024-04-24T23:43:49.235755+0000","last_undegraded":"2024-04-24T23:43:49.235755+0000","last_fullsized":"2024-04-24T23:43:49.235755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:12.034199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220193+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:43:49.220193+0000","last_peered":"2024-04-24T23:43:49.220193+0000","last_clean":"2024-04-24T23:43:49.220193+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:43:49.220193+0000","last_undegraded":"2024-04-24T23:43:49.220193+0000","last_fullsized":"2024-04-24T23:43:49.220193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224203+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:43:49.224203+0000","last_peered":"2024-04-24T23:43:49.224203+0000","last_clean":"2024-04-24T23:43:49.224203+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:43:49.224203+0000","last_undegraded":"2024-04-24T23:43:49.224203+0000","last_fullsized":"2024-04-24T23:43:49.224203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.236439+0000","last_change":"2024-04-24T23:43:49.236439+0000","last_active":"2024-04-24T23:43:49.236439+0000","last_peered":"2024-04-24T23:43:49.236439+0000","last_clean":"2024-04-24T23:43:49.236439+0000","last_became_active":"2024-04-24T23:43:49.227756+0000","last_became_peered":"2024-04-24T23:43:49.227756+0000","last_unstale":"2024-04-24T23:43:49.236439+0000","last_undegraded":"2024-04-24T23:43:49.236439+0000","last_fullsized":"2024-04-24T23:43:49.236439+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:34.790165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224250+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:43:49.224250+0000","last_peered":"2024-04-24T23:43:49.224250+0000","last_clean":"2024-04-24T23:43:49.224250+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:43:49.224250+0000","last_undegraded":"2024-04-24T23:43:49.224250+0000","last_fullsized":"2024-04-24T23:43:49.224250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216607+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:43:49.216607+0000","last_peered":"2024-04-24T23:43:49.216607+0000","last_clean":"2024-04-24T23:43:49.216607+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:43:49.216607+0000","last_undegraded":"2024-04-24T23:43:49.216607+0000","last_fullsized":"2024-04-24T23:43:49.216607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217594+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:43:49.217594+0000","last_peered":"2024-04-24T23:43:49.217594+0000","last_clean":"2024-04-24T23:43:49.217594+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:43:49.217594+0000","last_undegraded":"2024-04-24T23:43:49.217594+0000","last_fullsized":"2024-04-24T23:43:49.217594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.229019+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:43:49.229019+0000","last_peered":"2024-04-24T23:43:49.229019+0000","last_clean":"2024-04-24T23:43:49.229019+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:43:49.229019+0000","last_undegraded":"2024-04-24T23:43:49.229019+0000","last_fullsized":"2024-04-24T23:43:49.229019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217337+0000","last_change":"2024-04-24T23:42:55.322284+0000","last_active":"2024-04-24T23:43:49.217337+0000","last_peered":"2024-04-24T23:43:49.217337+0000","last_clean":"2024-04-24T23:43:49.217337+0000","last_became_active":"2024-04-24T23:42:55.321228+0000","last_became_peered":"2024-04-24T23:42:55.321228+0000","last_unstale":"2024-04-24T23:43:49.217337+0000","last_undegraded":"2024-04-24T23:43:49.217337+0000","last_fullsized":"2024-04-24T23:43:49.217337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:20.864377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221353+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:43:49.221353+0000","last_peered":"2024-04-24T23:43:49.221353+0000","last_clean":"2024-04-24T23:43:49.221353+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:43:49.221353+0000","last_undegraded":"2024-04-24T23:43:49.221353+0000","last_fullsized":"2024-04-24T23:43:49.221353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220416+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:43:49.220416+0000","last_peered":"2024-04-24T23:43:49.220416+0000","last_clean":"2024-04-24T23:43:49.220416+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:43:49.220416+0000","last_undegraded":"2024-04-24T23:43:49.220416+0000","last_fullsized":"2024-04-24T23:43:49.220416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220634+0000","last_change":"2024-04-24T23:42:55.324683+0000","last_active":"2024-04-24T23:43:49.220634+0000","last_peered":"2024-04-24T23:43:49.220634+0000","last_clean":"2024-04-24T23:43:49.220634+0000","last_became_active":"2024-04-24T23:42:55.324311+0000","last_became_peered":"2024-04-24T23:42:55.324311+0000","last_unstale":"2024-04-24T23:43:49.220634+0000","last_undegraded":"2024-04-24T23:43:49.220634+0000","last_fullsized":"2024-04-24T23:43:49.220634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:17.659811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223692+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:43:49.223692+0000","last_peered":"2024-04-24T23:43:49.223692+0000","last_clean":"2024-04-24T23:43:49.223692+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:43:49.223692+0000","last_undegraded":"2024-04-24T23:43:49.223692+0000","last_fullsized":"2024-04-24T23:43:49.223692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.222941+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:43:49.222941+0000","last_peered":"2024-04-24T23:43:49.222941+0000","last_clean":"2024-04-24T23:43:49.222941+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:43:49.222941+0000","last_undegraded":"2024-04-24T23:43:49.222941+0000","last_fullsized":"2024-04-24T23:43:49.222941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217209+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:43:49.217209+0000","last_peered":"2024-04-24T23:43:49.217209+0000","last_clean":"2024-04-24T23:43:49.217209+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:43:49.217209+0000","last_undegraded":"2024-04-24T23:43:49.217209+0000","last_fullsized":"2024-04-24T23:43:49.217209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216700+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:43:49.216700+0000","last_peered":"2024-04-24T23:43:49.216700+0000","last_clean":"2024-04-24T23:43:49.216700+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:43:49.216700+0000","last_undegraded":"2024-04-24T23:43:49.216700+0000","last_fullsized":"2024-04-24T23:43:49.216700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224350+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:43:49.224350+0000","last_peered":"2024-04-24T23:43:49.224350+0000","last_clean":"2024-04-24T23:43:49.224350+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:43:49.224350+0000","last_undegraded":"2024-04-24T23:43:49.224350+0000","last_fullsized":"2024-04-24T23:43:49.224350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217241+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:43:49.217241+0000","last_peered":"2024-04-24T23:43:49.217241+0000","last_clean":"2024-04-24T23:43:49.217241+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:43:49.217241+0000","last_undegraded":"2024-04-24T23:43:49.217241+0000","last_fullsized":"2024-04-24T23:43:49.217241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219789+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:43:49.219789+0000","last_peered":"2024-04-24T23:43:49.219789+0000","last_clean":"2024-04-24T23:43:49.219789+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:43:49.219789+0000","last_undegraded":"2024-04-24T23:43:49.219789+0000","last_fullsized":"2024-04-24T23:43:49.219789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218886+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:43:49.218886+0000","last_peered":"2024-04-24T23:43:49.218886+0000","last_clean":"2024-04-24T23:43:49.218886+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:43:49.218886+0000","last_undegraded":"2024-04-24T23:43:49.218886+0000","last_fullsized":"2024-04-24T23:43:49.218886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221316+0000","last_change":"2024-04-24T23:42:55.319168+0000","last_active":"2024-04-24T23:43:49.221316+0000","last_peered":"2024-04-24T23:43:49.221316+0000","last_clean":"2024-04-24T23:43:49.221316+0000","last_became_active":"2024-04-24T23:42:55.318752+0000","last_became_peered":"2024-04-24T23:42:55.318752+0000","last_unstale":"2024-04-24T23:43:49.221316+0000","last_undegraded":"2024-04-24T23:43:49.221316+0000","last_fullsized":"2024-04-24T23:43:49.221316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:14.923700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:28.228631+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:44:28.228631+0000","last_peered":"2024-04-24T23:44:28.228631+0000","last_clean":"2024-04-24T23:44:28.228631+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:44:28.228631+0000","last_undegraded":"2024-04-24T23:44:28.228631+0000","last_fullsized":"2024-04-24T23:44:28.228631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219422+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:43:49.219422+0000","last_peered":"2024-04-24T23:43:49.219422+0000","last_clean":"2024-04-24T23:43:49.219422+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:43:49.219422+0000","last_undegraded":"2024-04-24T23:43:49.219422+0000","last_fullsized":"2024-04-24T23:43:49.219422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228588+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:43:49.228588+0000","last_peered":"2024-04-24T23:43:49.228588+0000","last_clean":"2024-04-24T23:43:49.228588+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:43:49.228588+0000","last_undegraded":"2024-04-24T23:43:49.228588+0000","last_fullsized":"2024-04-24T23:43:49.228588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220427+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:43:49.220427+0000","last_peered":"2024-04-24T23:43:49.220427+0000","last_clean":"2024-04-24T23:43:49.220427+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:43:49.220427+0000","last_undegraded":"2024-04-24T23:43:49.220427+0000","last_fullsized":"2024-04-24T23:43:49.220427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219375+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:43:49.219375+0000","last_peered":"2024-04-24T23:43:49.219375+0000","last_clean":"2024-04-24T23:43:49.219375+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:43:49.219375+0000","last_undegraded":"2024-04-24T23:43:49.219375+0000","last_fullsized":"2024-04-24T23:43:49.219375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220798+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:43:49.220798+0000","last_peered":"2024-04-24T23:43:49.220798+0000","last_clean":"2024-04-24T23:43:49.220798+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:43:49.220798+0000","last_undegraded":"2024-04-24T23:43:49.220798+0000","last_fullsized":"2024-04-24T23:43:49.220798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220501+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:43:49.220501+0000","last_peered":"2024-04-24T23:43:49.220501+0000","last_clean":"2024-04-24T23:43:49.220501+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:43:49.220501+0000","last_undegraded":"2024-04-24T23:43:49.220501+0000","last_fullsized":"2024-04-24T23:43:49.220501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219482+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:43:49.219482+0000","last_peered":"2024-04-24T23:43:49.219482+0000","last_clean":"2024-04-24T23:43:49.219482+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:43:49.219482+0000","last_undegraded":"2024-04-24T23:43:49.219482+0000","last_fullsized":"2024-04-24T23:43:49.219482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223050+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:43:49.223050+0000","last_peered":"2024-04-24T23:43:49.223050+0000","last_clean":"2024-04-24T23:43:49.223050+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:43:49.223050+0000","last_undegraded":"2024-04-24T23:43:49.223050+0000","last_fullsized":"2024-04-24T23:43:49.223050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223272+0000","last_change":"2024-04-24T23:43:49.223272+0000","last_active":"2024-04-24T23:43:49.223272+0000","last_peered":"2024-04-24T23:43:49.223272+0000","last_clean":"2024-04-24T23:43:49.223272+0000","last_became_active":"2024-04-24T23:43:49.222921+0000","last_became_peered":"2024-04-24T23:43:49.222921+0000","last_unstale":"2024-04-24T23:43:49.223272+0000","last_undegraded":"2024-04-24T23:43:49.223272+0000","last_fullsized":"2024-04-24T23:43:49.223272+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:10.443359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217292+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:43:49.217292+0000","last_peered":"2024-04-24T23:43:49.217292+0000","last_clean":"2024-04-24T23:43:49.217292+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:43:49.217292+0000","last_undegraded":"2024-04-24T23:43:49.217292+0000","last_fullsized":"2024-04-24T23:43:49.217292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228726+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:43:49.228726+0000","last_peered":"2024-04-24T23:43:49.228726+0000","last_clean":"2024-04-24T23:43:49.228726+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:43:49.228726+0000","last_undegraded":"2024-04-24T23:43:49.228726+0000","last_fullsized":"2024-04-24T23:43:49.228726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.230897+0000","last_change":"2024-04-24T23:43:49.230897+0000","last_active":"2024-04-24T23:43:49.230897+0000","last_peered":"2024-04-24T23:43:49.230897+0000","last_clean":"2024-04-24T23:43:49.230897+0000","last_became_active":"2024-04-24T23:43:49.230514+0000","last_became_peered":"2024-04-24T23:43:49.230514+0000","last_unstale":"2024-04-24T23:43:49.230897+0000","last_undegraded":"2024-04-24T23:43:49.230897+0000","last_fullsized":"2024-04-24T23:43:49.230897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:24.826152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224781+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:43:49.224781+0000","last_peered":"2024-04-24T23:43:49.224781+0000","last_clean":"2024-04-24T23:43:49.224781+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:43:49.224781+0000","last_undegraded":"2024-04-24T23:43:49.224781+0000","last_fullsized":"2024-04-24T23:43:49.224781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217641+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:43:49.217641+0000","last_peered":"2024-04-24T23:43:49.217641+0000","last_clean":"2024-04-24T23:43:49.217641+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:43:49.217641+0000","last_undegraded":"2024-04-24T23:43:49.217641+0000","last_fullsized":"2024-04-24T23:43:49.217641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224407+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:43:49.224407+0000","last_peered":"2024-04-24T23:43:49.224407+0000","last_clean":"2024-04-24T23:43:49.224407+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:43:49.224407+0000","last_undegraded":"2024-04-24T23:43:49.224407+0000","last_fullsized":"2024-04-24T23:43:49.224407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221008+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:43:49.221008+0000","last_peered":"2024-04-24T23:43:49.221008+0000","last_clean":"2024-04-24T23:43:49.221008+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:43:49.221008+0000","last_undegraded":"2024-04-24T23:43:49.221008+0000","last_fullsized":"2024-04-24T23:43:49.221008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224036+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:43:49.224036+0000","last_peered":"2024-04-24T23:43:49.224036+0000","last_clean":"2024-04-24T23:43:49.224036+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:43:49.224036+0000","last_undegraded":"2024-04-24T23:43:49.224036+0000","last_fullsized":"2024-04-24T23:43:49.224036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221640+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:43:49.221640+0000","last_peered":"2024-04-24T23:43:49.221640+0000","last_clean":"2024-04-24T23:43:49.221640+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:43:49.221640+0000","last_undegraded":"2024-04-24T23:43:49.221640+0000","last_fullsized":"2024-04-24T23:43:49.221640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220109+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:43:49.220109+0000","last_peered":"2024-04-24T23:43:49.220109+0000","last_clean":"2024-04-24T23:43:49.220109+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:43:49.220109+0000","last_undegraded":"2024-04-24T23:43:49.220109+0000","last_fullsized":"2024-04-24T23:43:49.220109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223256+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:43:49.223256+0000","last_peered":"2024-04-24T23:43:49.223256+0000","last_clean":"2024-04-24T23:43:49.223256+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:43:49.223256+0000","last_undegraded":"2024-04-24T23:43:49.223256+0000","last_fullsized":"2024-04-24T23:43:49.223256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69146,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":722,"num_read_kb":431,"num_write":241,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":112,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":207438,"data_compressed":4866,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":501545,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1150973,"data_compressed":58446,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":501629,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":1098238,"data_compressed":56824,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":560230,"data_compressed":22260,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":560450,"data_compressed":22260,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56824,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":64085,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64305,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64460,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:44:32.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T23:44:33.019 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:33 smithi195 ceph-mon[33019]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:33.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:33 smithi195 ceph-mon[33019]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-24T23:44:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:33 smithi170 ceph-mon[32714]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:33.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:33 smithi170 ceph-mon[32714]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-24T23:44:33.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:33 smithi107 ceph-mon[29118]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:33.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:33 smithi107 ceph-mon[29118]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 -- 172.21.15.107:0/3361161168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd46006aee0 msgr2=0x7fd46006b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 --2- 172.21.15.107:0/3361161168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd46006aee0 0x7fd46006b360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd450009990 tx=0x7fd450009ca0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 -- 172.21.15.107:0/3361161168 shutdown_connections 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 --2- 172.21.15.107:0/3361161168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 0x7fd460105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 --2- 172.21.15.107:0/3361161168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd460107180 0x7fd46006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 --2- 172.21.15.107:0/3361161168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd46006aee0 0x7fd46006b360 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 -- 172.21.15.107:0/3361161168 >> 172.21.15.107:0/3361161168 conn(0x7fd4600771a0 msgr2=0x7fd4600775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 -- 172.21.15.107:0/3361161168 shutdown_connections 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.823+0000 7fd467502700 1 -- 172.21.15.107:0/3361161168 wait complete. 2024-04-24T23:44:34.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 Processor -- start 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 -- start start 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd46006aee0 0x7fd46019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 0x7fd46019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd460118b30 con 0x7fd460107180 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601189b0 con 0x7fd46006aee0 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd467502700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd460118cb0 con 0x7fd460104ca0 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60236/0 (socket says 172.21.15.107:60236) 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 -- 172.21.15.107:0/3784815962 learned_addr learned my addr 172.21.15.107:0/3784815962 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd464a9d700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 0x7fd46019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd46529e700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd46006aee0 0x7fd46019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:34.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 -- 172.21.15.107:0/3784815962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 msgr2=0x7fd46019f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 0x7fd46019f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 -- 172.21.15.107:0/3784815962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd46006aee0 msgr2=0x7fd46019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd46006aee0 0x7fd46019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.824+0000 7fd465a9f700 1 -- 172.21.15.107:0/3784815962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4601a3fd0 con 0x7fd460107180 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd465a9f700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd458007f10 tx=0x7fd45800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45800dbd0 con 0x7fd460107180 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd45800dd70 con 0x7fd460107180 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4601a42c0 con 0x7fd460107180 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4580133f0 con 0x7fd460107180 2024-04-24T23:44:34.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.825+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4601abb40 con 0x7fd460107180 2024-04-24T23:44:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.826+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd460101780 con 0x7fd460107180 2024-04-24T23:44:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.827+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd45801a020 con 0x7fd460107180 2024-04-24T23:44:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.827+0000 7fd4567fc700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 0x7fd44c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.827+0000 7fd46529e700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 0x7fd44c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:34.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.828+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd4580cf310 con 0x7fd460107180 2024-04-24T23:44:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.828+0000 7fd46529e700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 0x7fd44c0a51f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd45c006fd0 tx=0x7fd45c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:34.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:34.831+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd458010070 con 0x7fd460107180 2024-04-24T23:44:35.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.022+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd460104390 con 0x7fd44c0a2d30 2024-04-24T23:44:35.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.031+0000 7fd4567fc700 1 -- 172.21.15.107:0/3784815962 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340702 (secure 0 0 0) 0x7fd460104390 con 0x7fd44c0a2d30 2024-04-24T23:44:35.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:35.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 msgr2=0x7fd44c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:35.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 0x7fd44c0a51f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd45c006fd0 tx=0x7fd45c008040 comp rx=0 tx=0).stop 2024-04-24T23:44:35.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 msgr2=0x7fd4601a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:35.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd458007f10 tx=0x7fd45800d3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:35.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 shutdown_connections 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd44c0a2d30 0x7fd44c0a51f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd460107180 0x7fd4601a3890 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd46006aee0 0x7fd46019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.035+0000 7fd467502700 1 --2- 172.21.15.107:0/3784815962 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd460104ca0 0x7fd46019f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.036+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 >> 172.21.15.107:0/3784815962 conn(0x7fd4600771a0 msgr2=0x7fd460103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.036+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 shutdown_connections 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:35.036+0000 7fd467502700 1 -- 172.21.15.107:0/3784815962 wait complete. 2024-04-24T23:44:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:44:35.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:35 smithi170 ceph-mon[32714]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-24T23:44:35.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:35 smithi107 ceph-mon[29118]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-24T23:44:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:35 smithi195 ceph-mon[33019]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-24T23:44:35.875 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-24T23:44:34.000795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659748,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":878,"num_read_kb":560,"num_write":424,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1180848,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365680,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576144,"statfs":{"total":1151940427776,"available":1151565971456,"internally_reserved":0,"allocated":12591104,"data_stored":7965050,"data_compressed":361080,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":49,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223282+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:43:49.223282+0000","last_peered":"2024-04-24T23:43:49.223282+0000","last_clean":"2024-04-24T23:43:49.223282+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:43:49.223282+0000","last_undegraded":"2024-04-24T23:43:49.223282+0000","last_fullsized":"2024-04-24T23:43:49.223282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228886+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:43:49.228886+0000","last_peered":"2024-04-24T23:43:49.228886+0000","last_clean":"2024-04-24T23:43:49.228886+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:43:49.228886+0000","last_undegraded":"2024-04-24T23:43:49.228886+0000","last_fullsized":"2024-04-24T23:43:49.228886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228134+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:43:49.228134+0000","last_peered":"2024-04-24T23:43:49.228134+0000","last_clean":"2024-04-24T23:43:49.228134+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:43:49.228134+0000","last_undegraded":"2024-04-24T23:43:49.228134+0000","last_fullsized":"2024-04-24T23:43:49.228134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217676+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:43:49.217676+0000","last_peered":"2024-04-24T23:43:49.217676+0000","last_clean":"2024-04-24T23:43:49.217676+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:43:49.217676+0000","last_undegraded":"2024-04-24T23:43:49.217676+0000","last_fullsized":"2024-04-24T23:43:49.217676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219695+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:43:49.219695+0000","last_peered":"2024-04-24T23:43:49.219695+0000","last_clean":"2024-04-24T23:43:49.219695+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:43:49.219695+0000","last_undegraded":"2024-04-24T23:43:49.219695+0000","last_fullsized":"2024-04-24T23:43:49.219695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274694+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:43:49.274694+0000","last_peered":"2024-04-24T23:43:49.274694+0000","last_clean":"2024-04-24T23:43:49.274694+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:43:49.274694+0000","last_undegraded":"2024-04-24T23:43:49.274694+0000","last_fullsized":"2024-04-24T23:43:49.274694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218872+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:43:49.218872+0000","last_peered":"2024-04-24T23:43:49.218872+0000","last_clean":"2024-04-24T23:43:49.218872+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:43:49.218872+0000","last_undegraded":"2024-04-24T23:43:49.218872+0000","last_fullsized":"2024-04-24T23:43:49.218872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224087+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:43:49.224087+0000","last_peered":"2024-04-24T23:43:49.224087+0000","last_clean":"2024-04-24T23:43:49.224087+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:43:49.224087+0000","last_undegraded":"2024-04-24T23:43:49.224087+0000","last_fullsized":"2024-04-24T23:43:49.224087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228226+0000","last_change":"2024-04-24T23:43:49.227407+0000","last_active":"2024-04-24T23:43:49.228226+0000","last_peered":"2024-04-24T23:43:49.228226+0000","last_clean":"2024-04-24T23:43:49.228226+0000","last_became_active":"2024-04-24T23:43:49.227076+0000","last_became_peered":"2024-04-24T23:43:49.227076+0000","last_unstale":"2024-04-24T23:43:49.228226+0000","last_undegraded":"2024-04-24T23:43:49.228226+0000","last_fullsized":"2024-04-24T23:43:49.228226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:24.426521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.275170+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:43:49.275170+0000","last_peered":"2024-04-24T23:43:49.275170+0000","last_clean":"2024-04-24T23:43:49.275170+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:43:49.275170+0000","last_undegraded":"2024-04-24T23:43:49.275170+0000","last_fullsized":"2024-04-24T23:43:49.275170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219308+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:43:49.219308+0000","last_peered":"2024-04-24T23:43:49.219308+0000","last_clean":"2024-04-24T23:43:49.219308+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:43:49.219308+0000","last_undegraded":"2024-04-24T23:43:49.219308+0000","last_fullsized":"2024-04-24T23:43:49.219308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217511+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:43:49.217511+0000","last_peered":"2024-04-24T23:43:49.217511+0000","last_clean":"2024-04-24T23:43:49.217511+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:43:49.217511+0000","last_undegraded":"2024-04-24T23:43:49.217511+0000","last_fullsized":"2024-04-24T23:43:49.217511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220072+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:43:49.220072+0000","last_peered":"2024-04-24T23:43:49.220072+0000","last_clean":"2024-04-24T23:43:49.220072+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:43:49.220072+0000","last_undegraded":"2024-04-24T23:43:49.220072+0000","last_fullsized":"2024-04-24T23:43:49.220072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228190+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:43:49.228190+0000","last_peered":"2024-04-24T23:43:49.228190+0000","last_clean":"2024-04-24T23:43:49.228190+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:43:49.228190+0000","last_undegraded":"2024-04-24T23:43:49.228190+0000","last_fullsized":"2024-04-24T23:43:49.228190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228022+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:43:49.228022+0000","last_peered":"2024-04-24T23:43:49.228022+0000","last_clean":"2024-04-24T23:43:49.228022+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:43:49.228022+0000","last_undegraded":"2024-04-24T23:43:49.228022+0000","last_fullsized":"2024-04-24T23:43:49.228022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221339+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:43:49.221339+0000","last_peered":"2024-04-24T23:43:49.221339+0000","last_clean":"2024-04-24T23:43:49.221339+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:43:49.221339+0000","last_undegraded":"2024-04-24T23:43:49.221339+0000","last_fullsized":"2024-04-24T23:43:49.221339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223180+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:43:49.223180+0000","last_peered":"2024-04-24T23:43:49.223180+0000","last_clean":"2024-04-24T23:43:49.223180+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:43:49.223180+0000","last_undegraded":"2024-04-24T23:43:49.223180+0000","last_fullsized":"2024-04-24T23:43:49.223180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220249+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:43:49.220249+0000","last_peered":"2024-04-24T23:43:49.220249+0000","last_clean":"2024-04-24T23:43:49.220249+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:43:49.220249+0000","last_undegraded":"2024-04-24T23:43:49.220249+0000","last_fullsized":"2024-04-24T23:43:49.220249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217448+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:43:49.217448+0000","last_peered":"2024-04-24T23:43:49.217448+0000","last_clean":"2024-04-24T23:43:49.217448+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:43:49.217448+0000","last_undegraded":"2024-04-24T23:43:49.217448+0000","last_fullsized":"2024-04-24T23:43:49.217448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216955+0000","last_change":"2024-04-24T23:42:55.327166+0000","last_active":"2024-04-24T23:43:49.216955+0000","last_peered":"2024-04-24T23:43:49.216955+0000","last_clean":"2024-04-24T23:43:49.216955+0000","last_became_active":"2024-04-24T23:42:55.326708+0000","last_became_peered":"2024-04-24T23:42:55.326708+0000","last_unstale":"2024-04-24T23:43:49.216955+0000","last_undegraded":"2024-04-24T23:43:49.216955+0000","last_fullsized":"2024-04-24T23:43:49.216955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:39.604161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:29.194237+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:44:29.194237+0000","last_peered":"2024-04-24T23:44:29.194237+0000","last_clean":"2024-04-24T23:44:29.194237+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:44:29.194237+0000","last_undegraded":"2024-04-24T23:44:29.194237+0000","last_fullsized":"2024-04-24T23:44:29.194237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224207+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:43:49.224207+0000","last_peered":"2024-04-24T23:43:49.224207+0000","last_clean":"2024-04-24T23:43:49.224207+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:43:49.224207+0000","last_undegraded":"2024-04-24T23:43:49.224207+0000","last_fullsized":"2024-04-24T23:43:49.224207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217033+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:43:49.217033+0000","last_peered":"2024-04-24T23:43:49.217033+0000","last_clean":"2024-04-24T23:43:49.217033+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:43:49.217033+0000","last_undegraded":"2024-04-24T23:43:49.217033+0000","last_fullsized":"2024-04-24T23:43:49.217033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224563+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:43:49.224563+0000","last_peered":"2024-04-24T23:43:49.224563+0000","last_clean":"2024-04-24T23:43:49.224563+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:43:49.224563+0000","last_undegraded":"2024-04-24T23:43:49.224563+0000","last_fullsized":"2024-04-24T23:43:49.224563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228890+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:43:49.228890+0000","last_peered":"2024-04-24T23:43:49.228890+0000","last_clean":"2024-04-24T23:43:49.228890+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:43:49.228890+0000","last_undegraded":"2024-04-24T23:43:49.228890+0000","last_fullsized":"2024-04-24T23:43:49.228890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217145+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:43:49.217145+0000","last_peered":"2024-04-24T23:43:49.217145+0000","last_clean":"2024-04-24T23:43:49.217145+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:43:49.217145+0000","last_undegraded":"2024-04-24T23:43:49.217145+0000","last_fullsized":"2024-04-24T23:43:49.217145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.235090+0000","last_change":"2024-04-24T23:43:49.235090+0000","last_active":"2024-04-24T23:43:49.235090+0000","last_peered":"2024-04-24T23:43:49.235090+0000","last_clean":"2024-04-24T23:43:49.235090+0000","last_became_active":"2024-04-24T23:43:49.223773+0000","last_became_peered":"2024-04-24T23:43:49.223773+0000","last_unstale":"2024-04-24T23:43:49.235090+0000","last_undegraded":"2024-04-24T23:43:49.235090+0000","last_fullsized":"2024-04-24T23:43:49.235090+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:23.776176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221725+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:43:49.221725+0000","last_peered":"2024-04-24T23:43:49.221725+0000","last_clean":"2024-04-24T23:43:49.221725+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:43:49.221725+0000","last_undegraded":"2024-04-24T23:43:49.221725+0000","last_fullsized":"2024-04-24T23:43:49.221725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217650+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:43:49.217650+0000","last_peered":"2024-04-24T23:43:49.217650+0000","last_clean":"2024-04-24T23:43:49.217650+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:43:49.217650+0000","last_undegraded":"2024-04-24T23:43:49.217650+0000","last_fullsized":"2024-04-24T23:43:49.217650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219975+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:43:49.219975+0000","last_peered":"2024-04-24T23:43:49.219975+0000","last_clean":"2024-04-24T23:43:49.219975+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:43:49.219975+0000","last_undegraded":"2024-04-24T23:43:49.219975+0000","last_fullsized":"2024-04-24T23:43:49.219975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220338+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:43:49.220338+0000","last_peered":"2024-04-24T23:43:49.220338+0000","last_clean":"2024-04-24T23:43:49.220338+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:43:49.220338+0000","last_undegraded":"2024-04-24T23:43:49.220338+0000","last_fullsized":"2024-04-24T23:43:49.220338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216846+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:43:49.216846+0000","last_peered":"2024-04-24T23:43:49.216846+0000","last_clean":"2024-04-24T23:43:49.216846+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:43:49.216846+0000","last_undegraded":"2024-04-24T23:43:49.216846+0000","last_fullsized":"2024-04-24T23:43:49.216846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:31.198249+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:44:31.198249+0000","last_peered":"2024-04-24T23:44:31.198249+0000","last_clean":"2024-04-24T23:44:31.198249+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:44:31.198249+0000","last_undegraded":"2024-04-24T23:44:31.198249+0000","last_fullsized":"2024-04-24T23:44:31.198249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224709+0000","last_change":"2024-04-24T23:43:49.224709+0000","last_active":"2024-04-24T23:43:49.224709+0000","last_peered":"2024-04-24T23:43:49.224709+0000","last_clean":"2024-04-24T23:43:49.224709+0000","last_became_active":"2024-04-24T23:43:49.224345+0000","last_became_peered":"2024-04-24T23:43:49.224345+0000","last_unstale":"2024-04-24T23:43:49.224709+0000","last_undegraded":"2024-04-24T23:43:49.224709+0000","last_fullsized":"2024-04-24T23:43:49.224709+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:56.093532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216961+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:43:49.216961+0000","last_peered":"2024-04-24T23:43:49.216961+0000","last_clean":"2024-04-24T23:43:49.216961+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:43:49.216961+0000","last_undegraded":"2024-04-24T23:43:49.216961+0000","last_fullsized":"2024-04-24T23:43:49.216961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274646+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:43:49.274646+0000","last_peered":"2024-04-24T23:43:49.274646+0000","last_clean":"2024-04-24T23:43:49.274646+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:43:49.274646+0000","last_undegraded":"2024-04-24T23:43:49.274646+0000","last_fullsized":"2024-04-24T23:43:49.274646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219945+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:43:49.219945+0000","last_peered":"2024-04-24T23:43:49.219945+0000","last_clean":"2024-04-24T23:43:49.219945+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:43:49.219945+0000","last_undegraded":"2024-04-24T23:43:49.219945+0000","last_fullsized":"2024-04-24T23:43:49.219945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223167+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:43:49.223167+0000","last_peered":"2024-04-24T23:43:49.223167+0000","last_clean":"2024-04-24T23:43:49.223167+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:43:49.223167+0000","last_undegraded":"2024-04-24T23:43:49.223167+0000","last_fullsized":"2024-04-24T23:43:49.223167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220307+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:43:49.220307+0000","last_peered":"2024-04-24T23:43:49.220307+0000","last_clean":"2024-04-24T23:43:49.220307+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:43:49.220307+0000","last_undegraded":"2024-04-24T23:43:49.220307+0000","last_fullsized":"2024-04-24T23:43:49.220307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.227375+0000","last_change":"2024-04-24T23:43:49.226164+0000","last_active":"2024-04-24T23:43:49.227375+0000","last_peered":"2024-04-24T23:43:49.227375+0000","last_clean":"2024-04-24T23:43:49.227375+0000","last_became_active":"2024-04-24T23:43:49.225806+0000","last_became_peered":"2024-04-24T23:43:49.225806+0000","last_unstale":"2024-04-24T23:43:49.227375+0000","last_undegraded":"2024-04-24T23:43:49.227375+0000","last_fullsized":"2024-04-24T23:43:49.227375+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:56.068947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228954+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:43:49.228954+0000","last_peered":"2024-04-24T23:43:49.228954+0000","last_clean":"2024-04-24T23:43:49.228954+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:43:49.228954+0000","last_undegraded":"2024-04-24T23:43:49.228954+0000","last_fullsized":"2024-04-24T23:43:49.228954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220952+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:43:49.220952+0000","last_peered":"2024-04-24T23:43:49.220952+0000","last_clean":"2024-04-24T23:43:49.220952+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:43:49.220952+0000","last_undegraded":"2024-04-24T23:43:49.220952+0000","last_fullsized":"2024-04-24T23:43:49.220952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219413+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:43:49.219413+0000","last_peered":"2024-04-24T23:43:49.219413+0000","last_clean":"2024-04-24T23:43:49.219413+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:43:49.219413+0000","last_undegraded":"2024-04-24T23:43:49.219413+0000","last_fullsized":"2024-04-24T23:43:49.219413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223595+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:43:49.223595+0000","last_peered":"2024-04-24T23:43:49.223595+0000","last_clean":"2024-04-24T23:43:49.223595+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:43:49.223595+0000","last_undegraded":"2024-04-24T23:43:49.223595+0000","last_fullsized":"2024-04-24T23:43:49.223595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:30.196309+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:44:30.196309+0000","last_peered":"2024-04-24T23:44:30.196309+0000","last_clean":"2024-04-24T23:44:30.196309+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:44:30.196309+0000","last_undegraded":"2024-04-24T23:44:30.196309+0000","last_fullsized":"2024-04-24T23:44:30.196309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221890+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:43:49.221890+0000","last_peered":"2024-04-24T23:43:49.221890+0000","last_clean":"2024-04-24T23:43:49.221890+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:43:49.221890+0000","last_undegraded":"2024-04-24T23:43:49.221890+0000","last_fullsized":"2024-04-24T23:43:49.221890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219304+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:43:49.219304+0000","last_peered":"2024-04-24T23:43:49.219304+0000","last_clean":"2024-04-24T23:43:49.219304+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:43:49.219304+0000","last_undegraded":"2024-04-24T23:43:49.219304+0000","last_fullsized":"2024-04-24T23:43:49.219304+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.226781+0000","last_change":"2024-04-24T23:43:49.226781+0000","last_active":"2024-04-24T23:43:49.226781+0000","last_peered":"2024-04-24T23:43:49.226781+0000","last_clean":"2024-04-24T23:43:49.226781+0000","last_became_active":"2024-04-24T23:43:49.222089+0000","last_became_peered":"2024-04-24T23:43:49.222089+0000","last_unstale":"2024-04-24T23:43:49.226781+0000","last_undegraded":"2024-04-24T23:43:49.226781+0000","last_fullsized":"2024-04-24T23:43:49.226781+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:57.224621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228886+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:43:49.228886+0000","last_peered":"2024-04-24T23:43:49.228886+0000","last_clean":"2024-04-24T23:43:49.228886+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:43:49.228886+0000","last_undegraded":"2024-04-24T23:43:49.228886+0000","last_fullsized":"2024-04-24T23:43:49.228886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217287+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:43:49.217287+0000","last_peered":"2024-04-24T23:43:49.217287+0000","last_clean":"2024-04-24T23:43:49.217287+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:43:49.217287+0000","last_undegraded":"2024-04-24T23:43:49.217287+0000","last_fullsized":"2024-04-24T23:43:49.217287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220237+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:43:49.220237+0000","last_peered":"2024-04-24T23:43:49.220237+0000","last_clean":"2024-04-24T23:43:49.220237+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:43:49.220237+0000","last_undegraded":"2024-04-24T23:43:49.220237+0000","last_fullsized":"2024-04-24T23:43:49.220237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221866+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:43:49.221866+0000","last_peered":"2024-04-24T23:43:49.221866+0000","last_clean":"2024-04-24T23:43:49.221866+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:43:49.221866+0000","last_undegraded":"2024-04-24T23:43:49.221866+0000","last_fullsized":"2024-04-24T23:43:49.221866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219577+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:43:49.219577+0000","last_peered":"2024-04-24T23:43:49.219577+0000","last_clean":"2024-04-24T23:43:49.219577+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:43:49.219577+0000","last_undegraded":"2024-04-24T23:43:49.219577+0000","last_fullsized":"2024-04-24T23:43:49.219577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":212,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:30.196332+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:44:30.196332+0000","last_peered":"2024-04-24T23:44:30.196332+0000","last_clean":"2024-04-24T23:44:30.196332+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:44:30.196332+0000","last_undegraded":"2024-04-24T23:44:30.196332+0000","last_fullsized":"2024-04-24T23:44:30.196332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":136,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221537+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:43:49.221537+0000","last_peered":"2024-04-24T23:43:49.221537+0000","last_clean":"2024-04-24T23:43:49.221537+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:43:49.221537+0000","last_undegraded":"2024-04-24T23:43:49.221537+0000","last_fullsized":"2024-04-24T23:43:49.221537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220665+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:43:49.220665+0000","last_peered":"2024-04-24T23:43:49.220665+0000","last_clean":"2024-04-24T23:43:49.220665+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:43:49.220665+0000","last_undegraded":"2024-04-24T23:43:49.220665+0000","last_fullsized":"2024-04-24T23:43:49.220665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216901+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:43:49.216901+0000","last_peered":"2024-04-24T23:43:49.216901+0000","last_clean":"2024-04-24T23:43:49.216901+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:43:49.216901+0000","last_undegraded":"2024-04-24T23:43:49.216901+0000","last_fullsized":"2024-04-24T23:43:49.216901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219612+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:43:49.219612+0000","last_peered":"2024-04-24T23:43:49.219612+0000","last_clean":"2024-04-24T23:43:49.219612+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:43:49.219612+0000","last_undegraded":"2024-04-24T23:43:49.219612+0000","last_fullsized":"2024-04-24T23:43:49.219612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223064+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:43:49.223064+0000","last_peered":"2024-04-24T23:43:49.223064+0000","last_clean":"2024-04-24T23:43:49.223064+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:43:49.223064+0000","last_undegraded":"2024-04-24T23:43:49.223064+0000","last_fullsized":"2024-04-24T23:43:49.223064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219484+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:43:49.219484+0000","last_peered":"2024-04-24T23:43:49.219484+0000","last_clean":"2024-04-24T23:43:49.219484+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:43:49.219484+0000","last_undegraded":"2024-04-24T23:43:49.219484+0000","last_fullsized":"2024-04-24T23:43:49.219484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219229+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:43:49.219229+0000","last_peered":"2024-04-24T23:43:49.219229+0000","last_clean":"2024-04-24T23:43:49.219229+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:43:49.219229+0000","last_undegraded":"2024-04-24T23:43:49.219229+0000","last_fullsized":"2024-04-24T23:43:49.219229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220787+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:43:49.220787+0000","last_peered":"2024-04-24T23:43:49.220787+0000","last_clean":"2024-04-24T23:43:49.220787+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:43:49.220787+0000","last_undegraded":"2024-04-24T23:43:49.220787+0000","last_fullsized":"2024-04-24T23:43:49.220787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:27.192864+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:44:27.192864+0000","last_peered":"2024-04-24T23:44:27.192864+0000","last_clean":"2024-04-24T23:44:27.192864+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:44:27.192864+0000","last_undegraded":"2024-04-24T23:44:27.192864+0000","last_fullsized":"2024-04-24T23:44:27.192864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218968+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:43:49.218968+0000","last_peered":"2024-04-24T23:43:49.218968+0000","last_clean":"2024-04-24T23:43:49.218968+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:43:49.218968+0000","last_undegraded":"2024-04-24T23:43:49.218968+0000","last_fullsized":"2024-04-24T23:43:49.218968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220615+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:43:49.220615+0000","last_peered":"2024-04-24T23:43:49.220615+0000","last_clean":"2024-04-24T23:43:49.220615+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:43:49.220615+0000","last_undegraded":"2024-04-24T23:43:49.220615+0000","last_fullsized":"2024-04-24T23:43:49.220615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219577+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:43:49.219577+0000","last_peered":"2024-04-24T23:43:49.219577+0000","last_clean":"2024-04-24T23:43:49.219577+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:43:49.219577+0000","last_undegraded":"2024-04-24T23:43:49.219577+0000","last_fullsized":"2024-04-24T23:43:49.219577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223137+0000","last_change":"2024-04-24T23:43:49.223137+0000","last_active":"2024-04-24T23:43:49.223137+0000","last_peered":"2024-04-24T23:43:49.223137+0000","last_clean":"2024-04-24T23:43:49.223137+0000","last_became_active":"2024-04-24T23:43:49.222775+0000","last_became_peered":"2024-04-24T23:43:49.222775+0000","last_unstale":"2024-04-24T23:43:49.223137+0000","last_undegraded":"2024-04-24T23:43:49.223137+0000","last_fullsized":"2024-04-24T23:43:49.223137+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:22.910320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219413+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:43:49.219413+0000","last_peered":"2024-04-24T23:43:49.219413+0000","last_clean":"2024-04-24T23:43:49.219413+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:43:49.219413+0000","last_undegraded":"2024-04-24T23:43:49.219413+0000","last_fullsized":"2024-04-24T23:43:49.219413+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221934+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:43:49.221934+0000","last_peered":"2024-04-24T23:43:49.221934+0000","last_clean":"2024-04-24T23:43:49.221934+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:43:49.221934+0000","last_undegraded":"2024-04-24T23:43:49.221934+0000","last_fullsized":"2024-04-24T23:43:49.221934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224309+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:43:49.224309+0000","last_peered":"2024-04-24T23:43:49.224309+0000","last_clean":"2024-04-24T23:43:49.224309+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:43:49.224309+0000","last_undegraded":"2024-04-24T23:43:49.224309+0000","last_fullsized":"2024-04-24T23:43:49.224309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219939+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:43:49.219939+0000","last_peered":"2024-04-24T23:43:49.219939+0000","last_clean":"2024-04-24T23:43:49.219939+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:43:49.219939+0000","last_undegraded":"2024-04-24T23:43:49.219939+0000","last_fullsized":"2024-04-24T23:43:49.219939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220984+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:43:49.220984+0000","last_peered":"2024-04-24T23:43:49.220984+0000","last_clean":"2024-04-24T23:43:49.220984+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:43:49.220984+0000","last_undegraded":"2024-04-24T23:43:49.220984+0000","last_fullsized":"2024-04-24T23:43:49.220984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274963+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:43:49.274963+0000","last_peered":"2024-04-24T23:43:49.274963+0000","last_clean":"2024-04-24T23:43:49.274963+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:43:49.274963+0000","last_undegraded":"2024-04-24T23:43:49.274963+0000","last_fullsized":"2024-04-24T23:43:49.274963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220029+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:43:49.220029+0000","last_peered":"2024-04-24T23:43:49.220029+0000","last_clean":"2024-04-24T23:43:49.220029+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:43:49.220029+0000","last_undegraded":"2024-04-24T23:43:49.220029+0000","last_fullsized":"2024-04-24T23:43:49.220029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:28.193538+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:44:28.193538+0000","last_peered":"2024-04-24T23:44:28.193538+0000","last_clean":"2024-04-24T23:44:28.193538+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:44:28.193538+0000","last_undegraded":"2024-04-24T23:44:28.193538+0000","last_fullsized":"2024-04-24T23:44:28.193538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.274749+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:43:49.274749+0000","last_peered":"2024-04-24T23:43:49.274749+0000","last_clean":"2024-04-24T23:43:49.274749+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:43:49.274749+0000","last_undegraded":"2024-04-24T23:43:49.274749+0000","last_fullsized":"2024-04-24T23:43:49.274749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220729+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:43:49.220729+0000","last_peered":"2024-04-24T23:43:49.220729+0000","last_clean":"2024-04-24T23:43:49.220729+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:43:49.220729+0000","last_undegraded":"2024-04-24T23:43:49.220729+0000","last_fullsized":"2024-04-24T23:43:49.220729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221044+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:43:49.221044+0000","last_peered":"2024-04-24T23:43:49.221044+0000","last_clean":"2024-04-24T23:43:49.221044+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:43:49.221044+0000","last_undegraded":"2024-04-24T23:43:49.221044+0000","last_fullsized":"2024-04-24T23:43:49.221044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:27.192365+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:44:27.192365+0000","last_peered":"2024-04-24T23:44:27.192365+0000","last_clean":"2024-04-24T23:44:27.192365+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:44:27.192365+0000","last_undegraded":"2024-04-24T23:44:27.192365+0000","last_fullsized":"2024-04-24T23:44:27.192365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219192+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:43:49.219192+0000","last_peered":"2024-04-24T23:43:49.219192+0000","last_clean":"2024-04-24T23:43:49.219192+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:43:49.219192+0000","last_undegraded":"2024-04-24T23:43:49.219192+0000","last_fullsized":"2024-04-24T23:43:49.219192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223644+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:43:49.223644+0000","last_peered":"2024-04-24T23:43:49.223644+0000","last_clean":"2024-04-24T23:43:49.223644+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:43:49.223644+0000","last_undegraded":"2024-04-24T23:43:49.223644+0000","last_fullsized":"2024-04-24T23:43:49.223644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217258+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:43:49.217258+0000","last_peered":"2024-04-24T23:43:49.217258+0000","last_clean":"2024-04-24T23:43:49.217258+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:43:49.217258+0000","last_undegraded":"2024-04-24T23:43:49.217258+0000","last_fullsized":"2024-04-24T23:43:49.217258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218904+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:43:49.218904+0000","last_peered":"2024-04-24T23:43:49.218904+0000","last_clean":"2024-04-24T23:43:49.218904+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:43:49.218904+0000","last_undegraded":"2024-04-24T23:43:49.218904+0000","last_fullsized":"2024-04-24T23:43:49.218904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220890+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:43:49.220890+0000","last_peered":"2024-04-24T23:43:49.220890+0000","last_clean":"2024-04-24T23:43:49.220890+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:43:49.220890+0000","last_undegraded":"2024-04-24T23:43:49.220890+0000","last_fullsized":"2024-04-24T23:43:49.220890+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219005+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:43:49.219005+0000","last_peered":"2024-04-24T23:43:49.219005+0000","last_clean":"2024-04-24T23:43:49.219005+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:43:49.219005+0000","last_undegraded":"2024-04-24T23:43:49.219005+0000","last_fullsized":"2024-04-24T23:43:49.219005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218892+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:43:49.218892+0000","last_peered":"2024-04-24T23:43:49.218892+0000","last_clean":"2024-04-24T23:43:49.218892+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:43:49.218892+0000","last_undegraded":"2024-04-24T23:43:49.218892+0000","last_fullsized":"2024-04-24T23:43:49.218892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.222949+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:43:49.222949+0000","last_peered":"2024-04-24T23:43:49.222949+0000","last_clean":"2024-04-24T23:43:49.222949+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:43:49.222949+0000","last_undegraded":"2024-04-24T23:43:49.222949+0000","last_fullsized":"2024-04-24T23:43:49.222949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.235755+0000","last_change":"2024-04-24T23:43:49.235755+0000","last_active":"2024-04-24T23:43:49.235755+0000","last_peered":"2024-04-24T23:43:49.235755+0000","last_clean":"2024-04-24T23:43:49.235755+0000","last_became_active":"2024-04-24T23:43:49.230488+0000","last_became_peered":"2024-04-24T23:43:49.230488+0000","last_unstale":"2024-04-24T23:43:49.235755+0000","last_undegraded":"2024-04-24T23:43:49.235755+0000","last_fullsized":"2024-04-24T23:43:49.235755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:12.034199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220193+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:43:49.220193+0000","last_peered":"2024-04-24T23:43:49.220193+0000","last_clean":"2024-04-24T23:43:49.220193+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:43:49.220193+0000","last_undegraded":"2024-04-24T23:43:49.220193+0000","last_fullsized":"2024-04-24T23:43:49.220193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224203+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:43:49.224203+0000","last_peered":"2024-04-24T23:43:49.224203+0000","last_clean":"2024-04-24T23:43:49.224203+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:43:49.224203+0000","last_undegraded":"2024-04-24T23:43:49.224203+0000","last_fullsized":"2024-04-24T23:43:49.224203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.236439+0000","last_change":"2024-04-24T23:43:49.236439+0000","last_active":"2024-04-24T23:43:49.236439+0000","last_peered":"2024-04-24T23:43:49.236439+0000","last_clean":"2024-04-24T23:43:49.236439+0000","last_became_active":"2024-04-24T23:43:49.227756+0000","last_became_peered":"2024-04-24T23:43:49.227756+0000","last_unstale":"2024-04-24T23:43:49.236439+0000","last_undegraded":"2024-04-24T23:43:49.236439+0000","last_fullsized":"2024-04-24T23:43:49.236439+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:34.790165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224250+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:43:49.224250+0000","last_peered":"2024-04-24T23:43:49.224250+0000","last_clean":"2024-04-24T23:43:49.224250+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:43:49.224250+0000","last_undegraded":"2024-04-24T23:43:49.224250+0000","last_fullsized":"2024-04-24T23:43:49.224250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216607+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:43:49.216607+0000","last_peered":"2024-04-24T23:43:49.216607+0000","last_clean":"2024-04-24T23:43:49.216607+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:43:49.216607+0000","last_undegraded":"2024-04-24T23:43:49.216607+0000","last_fullsized":"2024-04-24T23:43:49.216607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217594+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:43:49.217594+0000","last_peered":"2024-04-24T23:43:49.217594+0000","last_clean":"2024-04-24T23:43:49.217594+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:43:49.217594+0000","last_undegraded":"2024-04-24T23:43:49.217594+0000","last_fullsized":"2024-04-24T23:43:49.217594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.229019+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:43:49.229019+0000","last_peered":"2024-04-24T23:43:49.229019+0000","last_clean":"2024-04-24T23:43:49.229019+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:43:49.229019+0000","last_undegraded":"2024-04-24T23:43:49.229019+0000","last_fullsized":"2024-04-24T23:43:49.229019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217337+0000","last_change":"2024-04-24T23:42:55.322284+0000","last_active":"2024-04-24T23:43:49.217337+0000","last_peered":"2024-04-24T23:43:49.217337+0000","last_clean":"2024-04-24T23:43:49.217337+0000","last_became_active":"2024-04-24T23:42:55.321228+0000","last_became_peered":"2024-04-24T23:42:55.321228+0000","last_unstale":"2024-04-24T23:43:49.217337+0000","last_undegraded":"2024-04-24T23:43:49.217337+0000","last_fullsized":"2024-04-24T23:43:49.217337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:20.864377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221353+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:43:49.221353+0000","last_peered":"2024-04-24T23:43:49.221353+0000","last_clean":"2024-04-24T23:43:49.221353+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:43:49.221353+0000","last_undegraded":"2024-04-24T23:43:49.221353+0000","last_fullsized":"2024-04-24T23:43:49.221353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220416+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:43:49.220416+0000","last_peered":"2024-04-24T23:43:49.220416+0000","last_clean":"2024-04-24T23:43:49.220416+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:43:49.220416+0000","last_undegraded":"2024-04-24T23:43:49.220416+0000","last_fullsized":"2024-04-24T23:43:49.220416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220634+0000","last_change":"2024-04-24T23:42:55.324683+0000","last_active":"2024-04-24T23:43:49.220634+0000","last_peered":"2024-04-24T23:43:49.220634+0000","last_clean":"2024-04-24T23:43:49.220634+0000","last_became_active":"2024-04-24T23:42:55.324311+0000","last_became_peered":"2024-04-24T23:42:55.324311+0000","last_unstale":"2024-04-24T23:43:49.220634+0000","last_undegraded":"2024-04-24T23:43:49.220634+0000","last_fullsized":"2024-04-24T23:43:49.220634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:17.659811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223692+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:43:49.223692+0000","last_peered":"2024-04-24T23:43:49.223692+0000","last_clean":"2024-04-24T23:43:49.223692+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:43:49.223692+0000","last_undegraded":"2024-04-24T23:43:49.223692+0000","last_fullsized":"2024-04-24T23:43:49.223692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.222941+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:43:49.222941+0000","last_peered":"2024-04-24T23:43:49.222941+0000","last_clean":"2024-04-24T23:43:49.222941+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:43:49.222941+0000","last_undegraded":"2024-04-24T23:43:49.222941+0000","last_fullsized":"2024-04-24T23:43:49.222941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217209+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:43:49.217209+0000","last_peered":"2024-04-24T23:43:49.217209+0000","last_clean":"2024-04-24T23:43:49.217209+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:43:49.217209+0000","last_undegraded":"2024-04-24T23:43:49.217209+0000","last_fullsized":"2024-04-24T23:43:49.217209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.216700+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:43:49.216700+0000","last_peered":"2024-04-24T23:43:49.216700+0000","last_clean":"2024-04-24T23:43:49.216700+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:43:49.216700+0000","last_undegraded":"2024-04-24T23:43:49.216700+0000","last_fullsized":"2024-04-24T23:43:49.216700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224350+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:43:49.224350+0000","last_peered":"2024-04-24T23:43:49.224350+0000","last_clean":"2024-04-24T23:43:49.224350+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:43:49.224350+0000","last_undegraded":"2024-04-24T23:43:49.224350+0000","last_fullsized":"2024-04-24T23:43:49.224350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217241+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:43:49.217241+0000","last_peered":"2024-04-24T23:43:49.217241+0000","last_clean":"2024-04-24T23:43:49.217241+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:43:49.217241+0000","last_undegraded":"2024-04-24T23:43:49.217241+0000","last_fullsized":"2024-04-24T23:43:49.217241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219789+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:43:49.219789+0000","last_peered":"2024-04-24T23:43:49.219789+0000","last_clean":"2024-04-24T23:43:49.219789+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:43:49.219789+0000","last_undegraded":"2024-04-24T23:43:49.219789+0000","last_fullsized":"2024-04-24T23:43:49.219789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.218886+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:43:49.218886+0000","last_peered":"2024-04-24T23:43:49.218886+0000","last_clean":"2024-04-24T23:43:49.218886+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:43:49.218886+0000","last_undegraded":"2024-04-24T23:43:49.218886+0000","last_fullsized":"2024-04-24T23:43:49.218886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221316+0000","last_change":"2024-04-24T23:42:55.319168+0000","last_active":"2024-04-24T23:43:49.221316+0000","last_peered":"2024-04-24T23:43:49.221316+0000","last_clean":"2024-04-24T23:43:49.221316+0000","last_became_active":"2024-04-24T23:42:55.318752+0000","last_became_peered":"2024-04-24T23:42:55.318752+0000","last_unstale":"2024-04-24T23:43:49.221316+0000","last_undegraded":"2024-04-24T23:43:49.221316+0000","last_fullsized":"2024-04-24T23:43:49.221316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:14.923700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:44:28.228631+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:44:28.228631+0000","last_peered":"2024-04-24T23:44:28.228631+0000","last_clean":"2024-04-24T23:44:28.228631+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:44:28.228631+0000","last_undegraded":"2024-04-24T23:44:28.228631+0000","last_fullsized":"2024-04-24T23:44:28.228631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219422+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:43:49.219422+0000","last_peered":"2024-04-24T23:43:49.219422+0000","last_clean":"2024-04-24T23:43:49.219422+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:43:49.219422+0000","last_undegraded":"2024-04-24T23:43:49.219422+0000","last_fullsized":"2024-04-24T23:43:49.219422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228588+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:43:49.228588+0000","last_peered":"2024-04-24T23:43:49.228588+0000","last_clean":"2024-04-24T23:43:49.228588+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:43:49.228588+0000","last_undegraded":"2024-04-24T23:43:49.228588+0000","last_fullsized":"2024-04-24T23:43:49.228588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220427+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:43:49.220427+0000","last_peered":"2024-04-24T23:43:49.220427+0000","last_clean":"2024-04-24T23:43:49.220427+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:43:49.220427+0000","last_undegraded":"2024-04-24T23:43:49.220427+0000","last_fullsized":"2024-04-24T23:43:49.220427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219375+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:43:49.219375+0000","last_peered":"2024-04-24T23:43:49.219375+0000","last_clean":"2024-04-24T23:43:49.219375+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:43:49.219375+0000","last_undegraded":"2024-04-24T23:43:49.219375+0000","last_fullsized":"2024-04-24T23:43:49.219375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220798+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:43:49.220798+0000","last_peered":"2024-04-24T23:43:49.220798+0000","last_clean":"2024-04-24T23:43:49.220798+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:43:49.220798+0000","last_undegraded":"2024-04-24T23:43:49.220798+0000","last_fullsized":"2024-04-24T23:43:49.220798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220501+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:43:49.220501+0000","last_peered":"2024-04-24T23:43:49.220501+0000","last_clean":"2024-04-24T23:43:49.220501+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:43:49.220501+0000","last_undegraded":"2024-04-24T23:43:49.220501+0000","last_fullsized":"2024-04-24T23:43:49.220501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.219482+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:43:49.219482+0000","last_peered":"2024-04-24T23:43:49.219482+0000","last_clean":"2024-04-24T23:43:49.219482+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:43:49.219482+0000","last_undegraded":"2024-04-24T23:43:49.219482+0000","last_fullsized":"2024-04-24T23:43:49.219482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223050+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:43:49.223050+0000","last_peered":"2024-04-24T23:43:49.223050+0000","last_clean":"2024-04-24T23:43:49.223050+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:43:49.223050+0000","last_undegraded":"2024-04-24T23:43:49.223050+0000","last_fullsized":"2024-04-24T23:43:49.223050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223272+0000","last_change":"2024-04-24T23:43:49.223272+0000","last_active":"2024-04-24T23:43:49.223272+0000","last_peered":"2024-04-24T23:43:49.223272+0000","last_clean":"2024-04-24T23:43:49.223272+0000","last_became_active":"2024-04-24T23:43:49.222921+0000","last_became_peered":"2024-04-24T23:43:49.222921+0000","last_unstale":"2024-04-24T23:43:49.223272+0000","last_undegraded":"2024-04-24T23:43:49.223272+0000","last_fullsized":"2024-04-24T23:43:49.223272+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:10.443359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217292+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:43:49.217292+0000","last_peered":"2024-04-24T23:43:49.217292+0000","last_clean":"2024-04-24T23:43:49.217292+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:43:49.217292+0000","last_undegraded":"2024-04-24T23:43:49.217292+0000","last_fullsized":"2024-04-24T23:43:49.217292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.228726+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:43:49.228726+0000","last_peered":"2024-04-24T23:43:49.228726+0000","last_clean":"2024-04-24T23:43:49.228726+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:43:49.228726+0000","last_undegraded":"2024-04-24T23:43:49.228726+0000","last_fullsized":"2024-04-24T23:43:49.228726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.230897+0000","last_change":"2024-04-24T23:43:49.230897+0000","last_active":"2024-04-24T23:43:49.230897+0000","last_peered":"2024-04-24T23:43:49.230897+0000","last_clean":"2024-04-24T23:43:49.230897+0000","last_became_active":"2024-04-24T23:43:49.230514+0000","last_became_peered":"2024-04-24T23:43:49.230514+0000","last_unstale":"2024-04-24T23:43:49.230897+0000","last_undegraded":"2024-04-24T23:43:49.230897+0000","last_fullsized":"2024-04-24T23:43:49.230897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:24.826152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224781+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:43:49.224781+0000","last_peered":"2024-04-24T23:43:49.224781+0000","last_clean":"2024-04-24T23:43:49.224781+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:43:49.224781+0000","last_undegraded":"2024-04-24T23:43:49.224781+0000","last_fullsized":"2024-04-24T23:43:49.224781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.217641+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:43:49.217641+0000","last_peered":"2024-04-24T23:43:49.217641+0000","last_clean":"2024-04-24T23:43:49.217641+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:43:49.217641+0000","last_undegraded":"2024-04-24T23:43:49.217641+0000","last_fullsized":"2024-04-24T23:43:49.217641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224407+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:43:49.224407+0000","last_peered":"2024-04-24T23:43:49.224407+0000","last_clean":"2024-04-24T23:43:49.224407+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:43:49.224407+0000","last_undegraded":"2024-04-24T23:43:49.224407+0000","last_fullsized":"2024-04-24T23:43:49.224407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221008+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:43:49.221008+0000","last_peered":"2024-04-24T23:43:49.221008+0000","last_clean":"2024-04-24T23:43:49.221008+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:43:49.221008+0000","last_undegraded":"2024-04-24T23:43:49.221008+0000","last_fullsized":"2024-04-24T23:43:49.221008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.224036+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:43:49.224036+0000","last_peered":"2024-04-24T23:43:49.224036+0000","last_clean":"2024-04-24T23:43:49.224036+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:43:49.224036+0000","last_undegraded":"2024-04-24T23:43:49.224036+0000","last_fullsized":"2024-04-24T23:43:49.224036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.221640+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:43:49.221640+0000","last_peered":"2024-04-24T23:43:49.221640+0000","last_clean":"2024-04-24T23:43:49.221640+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:43:49.221640+0000","last_undegraded":"2024-04-24T23:43:49.221640+0000","last_fullsized":"2024-04-24T23:43:49.221640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.220109+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:43:49.220109+0000","last_peered":"2024-04-24T23:43:49.220109+0000","last_clean":"2024-04-24T23:43:49.220109+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:43:49.220109+0000","last_undegraded":"2024-04-24T23:43:49.220109+0000","last_fullsized":"2024-04-24T23:43:49.220109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T23:43:49.223256+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:43:49.223256+0000","last_peered":"2024-04-24T23:43:49.223256+0000","last_clean":"2024-04-24T23:43:49.223256+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:43:49.223256+0000","last_undegraded":"2024-04-24T23:43:49.223256+0000","last_fullsized":"2024-04-24T23:43:49.223256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69146,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":752,"num_read_kb":451,"num_write":241,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":112,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":207438,"data_compressed":4866,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":501545,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1150973,"data_compressed":58446,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":501629,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":1098238,"data_compressed":56824,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":560230,"data_compressed":22260,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":560450,"data_compressed":22260,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20638,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56824,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":64085,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64305,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64460,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:44:35.881 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-24T23:44:35.881 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T23:44:35.881 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-24T23:44:35.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-24T23:44:36.123 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:36.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:36 smithi107 ceph-mon[29118]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:36.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:36 smithi170 ceph-mon[32714]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:36.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:36 smithi195 ceph-mon[33019]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:44:37.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:37 smithi170 ceph-mon[32714]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-24T23:44:37.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:37 smithi107 ceph-mon[29118]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-24T23:44:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:37 smithi195 ceph-mon[33019]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-24T23:44:37.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.965+0000 7f417b41c700 1 -- 172.21.15.107:0/1894011951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 msgr2=0x7f417406afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 --2- 172.21.15.107:0/1894011951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417406afe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f41700099c0 tx=0x7f4170009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 -- 172.21.15.107:0/1894011951 shutdown_connections 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 --2- 172.21.15.107:0/1894011951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 0x7f417410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 --2- 172.21.15.107:0/1894011951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417406afe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 --2- 172.21.15.107:0/1894011951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 0x7f4174106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 -- 172.21.15.107:0/1894011951 >> 172.21.15.107:0/1894011951 conn(0x7f41740771a0 msgr2=0x7f41740775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 -- 172.21.15.107:0/1894011951 shutdown_connections 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 -- 172.21.15.107:0/1894011951 wait complete. 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 Processor -- start 2024-04-24T23:44:37.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 -- start start 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.966+0000 7f417b41c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f417b41c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 0x7f417419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f417b41c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 0x7f41741987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f417b41c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112590 con 0x7f417406ac00 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f417b41c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112410 con 0x7f41741075e0 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f417b41c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4174112710 con 0x7f417406b5b0 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46880/0 (socket says 172.21.15.107:46880) 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 -- 172.21.15.107:0/1500171678 learned_addr learned my addr 172.21.15.107:0/1500171678 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41799b9700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 0x7f41741987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41789b7700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 0x7f417419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 -- 172.21.15.107:0/1500171678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 msgr2=0x7f417419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 0x7f417419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 -- 172.21.15.107:0/1500171678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 msgr2=0x7f41741987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 0x7f41741987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 -- 172.21.15.107:0/1500171678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4174198fd0 con 0x7f417406ac00 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f41791b8700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4170009fd0 tx=0x7f4170003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.967+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170027070 con 0x7f417406ac00 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.968+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4174199260 con 0x7f417406ac00 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.968+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4170004050 con 0x7f417406ac00 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.968+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170022db0 con 0x7f417406ac00 2024-04-24T23:44:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.968+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4174101160 con 0x7f417406ac00 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.968+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4158005330 con 0x7f417406ac00 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.970+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f417002a070 con 0x7f417406ac00 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.970+0000 7f416a7fc700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 0x7f41600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.970+0000 7f41789b7700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 0x7f41600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.970+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f41700d0ae0 con 0x7f417406ac00 2024-04-24T23:44:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.971+0000 7f41789b7700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 0x7f41600a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4174107010 tx=0x7f416400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:37.974+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4170031070 con 0x7f417406ac00 2024-04-24T23:44:38.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.217+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4158005620 con 0x7f417406ac00 2024-04-24T23:44:38.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.217+0000 7f416a7fc700 1 -- 172.21.15.107:0/1500171678 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4170013070 con 0x7f417406ac00 2024-04-24T23:44:38.217 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:38.217 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T23:44:38.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 msgr2=0x7f41600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:38.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 0x7f41600a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4174107010 tx=0x7f416400a300 comp rx=0 tx=0).stop 2024-04-24T23:44:38.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 msgr2=0x7f417419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:38.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4170009fd0 tx=0x7f4170003680 comp rx=0 tx=0).stop 2024-04-24T23:44:38.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 shutdown_connections 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f41600a2cb0 0x7f41600a5170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41741075e0 0x7f41741987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f417406ac00 0x7f417419e670 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 --2- 172.21.15.107:0/1500171678 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f417406b5b0 0x7f417419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.219+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 >> 172.21.15.107:0/1500171678 conn(0x7f41740771a0 msgr2=0x7f4174100900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.220+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 shutdown_connections 2024-04-24T23:44:38.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.220+0000 7f417b41c700 1 -- 172.21.15.107:0/1500171678 wait complete. 2024-04-24T23:44:38.882 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-24T23:44:38.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.983+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1432977168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 msgr2=0x7fe898101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.983+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1432977168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898101d00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009900 tx=0x7fe88c02a410 comp rx=0 tx=0).stop 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1432977168 shutdown_connections 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1432977168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898102240 0x7fe898108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1432977168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898101d00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1432977168 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe898100690 0x7fe898100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1432977168 >> 172.21.15.107:0/1432977168 conn(0x7fe8980fbc40 msgr2=0x7fe8980fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:38.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1432977168 shutdown_connections 2024-04-24T23:44:38.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1432977168 wait complete. 2024-04-24T23:44:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 Processor -- start 2024-04-24T23:44:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- start start 2024-04-24T23:44:38.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 0x7fe898197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe898102240 0x7fe89819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe89810c380 con 0x7fe898101880 2024-04-24T23:44:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe89810c200 con 0x7fe898100690 2024-04-24T23:44:38.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.984+0000 7fe89c8a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe89810c500 con 0x7fe898102240 2024-04-24T23:44:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46892/0 (socket says 172.21.15.107:46892) 2024-04-24T23:44:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 -- 172.21.15.107:0/1936809222 learned_addr learned my addr 172.21.15.107:0/1936809222 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe89659c640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 0x7fe898197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 -- 172.21.15.107:0/1936809222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe898102240 msgr2=0x7fe89819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe898102240 0x7fe89819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 -- 172.21.15.107:0/1936809222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 msgr2=0x7fe898197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 0x7fe898197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 -- 172.21.15.107:0/1936809222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89819f780 con 0x7fe898101880 2024-04-24T23:44:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe89659c640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 0x7fe898197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe895d9b640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009230 tx=0x7fe88c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c03f030 con 0x7fe898101880 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89819fa10 con 0x7fe898101880 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe88c003ce0 con 0x7fe898101880 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c035070 con 0x7fe898101880 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.985+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89819ff80 con 0x7fe898101880 2024-04-24T23:44:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.986+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8981a01b0 con 0x7fe898101880 2024-04-24T23:44:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.987+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe88c042030 con 0x7fe898101880 2024-04-24T23:44:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.988+0000 7fe8877fe640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 0x7fe8800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.988+0000 7fe89659c640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 0x7fe8800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.988+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe88c0f0140 con 0x7fe898101880 2024-04-24T23:44:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.988+0000 7fe89659c640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 0x7fe8800a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe888006fd0 tx=0x7fe888008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:38.992+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe88c0bcbe0 con 0x7fe898101880 2024-04-24T23:44:39.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:39 smithi107 ceph-mon[29118]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-24T23:44:39.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:39 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1500171678' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T23:44:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.169+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8981a0540 con 0x7fe898101880 2024-04-24T23:44:39.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.171+0000 7fe8877fe640 1 -- 172.21.15.107:0/1936809222 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24715 (secure 0 0 0) 0x7fe88c0bc4e0 con 0x7fe898101880 2024-04-24T23:44:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:39.172 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:43:49.212670+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:44:39.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 msgr2=0x7fe8800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 0x7fe8800a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe888006fd0 tx=0x7fe888008040 comp rx=0 tx=0).stop 2024-04-24T23:44:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 msgr2=0x7fe898198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009230 tx=0x7fe88c004430 comp rx=0 tx=0).stop 2024-04-24T23:44:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 shutdown_connections 2024-04-24T23:44:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe8800a2c20 0x7fe8800a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe898102240 0x7fe89819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe898101880 0x7fe898198110 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 --2- 172.21.15.107:0/1936809222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898100690 0x7fe898197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 >> 172.21.15.107:0/1936809222 conn(0x7fe8980fbc40 msgr2=0x7fe8980fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 shutdown_connections 2024-04-24T23:44:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.173+0000 7fe89c8a5640 1 -- 172.21.15.107:0/1936809222 wait complete. 2024-04-24T23:44:39.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:44:39.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 -- 172.21.15.107:0/3362846585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10107460 msgr2=0x7f5c101098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/3362846585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10107460 0x7f5c101098b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009900 tx=0x7f5c0802a410 comp rx=0 tx=0).stop 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 -- 172.21.15.107:0/3362846585 shutdown_connections 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/3362846585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10107460 0x7f5c101098b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/3362846585 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10104b30 0x7f5c10106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/3362846585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c101045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 -- 172.21.15.107:0/3362846585 >> 172.21.15.107:0/3362846585 conn(0x7f5c100fbbf0 msgr2=0x7f5c100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 -- 172.21.15.107:0/3362846585 shutdown_connections 2024-04-24T23:44:39.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.289+0000 7f5c17ee1640 1 -- 172.21.15.107:0/3362846585 wait complete. 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 Processor -- start 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 -- start start 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c10101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10107460 0x7f5c101006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010d420 con 0x7f5c10104b30 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010d2a0 con 0x7f5c10102200 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c17ee1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010d5a0 con 0x7f5c10107460 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c1547c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c1547c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46910/0 (socket says 172.21.15.107:46910) 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.290+0000 7f5c1547c640 1 -- 172.21.15.107:0/1707977438 learned_addr learned my addr 172.21.15.107:0/1707977438 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c15c7d640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c10101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1647e640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10107460 0x7f5c101006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 -- 172.21.15.107:0/1707977438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10107460 msgr2=0x7f5c101006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10107460 0x7f5c101006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 -- 172.21.15.107:0/1707977438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 msgr2=0x7f5c10101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c10101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 -- 172.21.15.107:0/1707977438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c10100f80 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c15c7d640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c10101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c1547c640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c970 tx=0x7f5c0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007bb0 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c00007d50 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c101a1be0 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c000187a0 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.291+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c101a20e0 con 0x7f5c10104b30 2024-04-24T23:44:39.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.292+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c100fd6e0 con 0x7f5c10104b30 2024-04-24T23:44:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.293+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5c0000f450 con 0x7f5c10104b30 2024-04-24T23:44:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.294+0000 7f5c06ffd640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 0x7f5bfc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.294+0000 7f5c15c7d640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 0x7f5bfc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.294+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5c000cf6e0 con 0x7f5c10104b30 2024-04-24T23:44:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.294+0000 7f5c15c7d640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 0x7f5bfc0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c005fd0 tx=0x7f5c0c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.298+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5c00014030 con 0x7f5c10104b30 2024-04-24T23:44:39.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:39 smithi170 ceph-mon[32714]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-24T23:44:39.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:39 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1500171678' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T23:44:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.477+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5c1002d390 con 0x7f5c10104b30 2024-04-24T23:44:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.477+0000 7f5c06ffd640 1 -- 172.21.15.107:0/1707977438 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f5c0009bb30 con 0x7f5c10104b30 2024-04-24T23:44:39.477 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 msgr2=0x7f5bfc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 0x7f5bfc0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c005fd0 tx=0x7f5c0c009510 comp rx=0 tx=0).stop 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 msgr2=0x7f5c101001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c970 tx=0x7f5c0000ce30 comp rx=0 tx=0).stop 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 shutdown_connections 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5bfc0a2c20 0x7f5bfc0a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5c10107460 0x7f5c101006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10104b30 0x7f5c101001b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 --2- 172.21.15.107:0/1707977438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c10102200 0x7f5c10101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 >> 172.21.15.107:0/1707977438 conn(0x7f5c100fbbf0 msgr2=0x7f5c10102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 shutdown_connections 2024-04-24T23:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.479+0000 7f5c17ee1640 1 -- 172.21.15.107:0/1707977438 wait complete. 2024-04-24T23:44:39.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:44:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:39 smithi195 ceph-mon[33019]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-24T23:44:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:39 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1500171678' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T23:44:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- 172.21.15.107:0/3114719821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104102980 msgr2=0x7f3104102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 --2- 172.21.15.107:0/3114719821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104102980 0x7f3104102e00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009900 tx=0x7f30f802a410 comp rx=0 tx=0).stop 2024-04-24T23:44:39.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- 172.21.15.107:0/3114719821 shutdown_connections 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 --2- 172.21.15.107:0/3114719821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f3104109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 --2- 172.21.15.107:0/3114719821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104102980 0x7f3104102e00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 --2- 172.21.15.107:0/3114719821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104101790 0x7f3104101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- 172.21.15.107:0/3114719821 >> 172.21.15.107:0/3114719821 conn(0x7f31040fcd20 msgr2=0x7f31040ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- 172.21.15.107:0/3114719821 shutdown_connections 2024-04-24T23:44:39.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- 172.21.15.107:0/3114719821 wait complete. 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 Processor -- start 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.591+0000 7f3109076640 1 -- start start 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104102980 0x7f3104197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f310419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f310410d440 con 0x7f3104101790 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f310410d2c0 con 0x7f3104103340 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f310410d5c0 con 0x7f3104102980 2024-04-24T23:44:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46924/0 (socket says 172.21.15.107:46924) 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 -- 172.21.15.107:0/2824986204 learned_addr learned my addr 172.21.15.107:0/2824986204 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f310259c640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104102980 0x7f3104197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f310359e640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f310419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 -- 172.21.15.107:0/2824986204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104102980 msgr2=0x7f3104197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104102980 0x7f3104197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 -- 172.21.15.107:0/2824986204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 msgr2=0x7f310419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f310419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 -- 172.21.15.107:0/2824986204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310419f6d0 con 0x7f3104101790 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f310359e640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f310419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3102d9d640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f30f400c970 tx=0x7f30f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4004220 con 0x7f3104101790 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310419f9c0 con 0x7f3104101790 2024-04-24T23:44:39.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30f40043c0 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310419ff30 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.592+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f401d060 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.593+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3104108ab0 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.594+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f30f40079b0 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.595+0000 7f30ebfff640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 0x7f30ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.595+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f30f40cf6b0 con 0x7f3104101790 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.595+0000 7f310259c640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 0x7f30ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.595+0000 7f310259c640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 0x7f30ec0a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30f8006010 tx=0x7f30f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.599+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f30f4014030 con 0x7f3104101790 2024-04-24T23:44:39.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.774+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3104062340 con 0x7f3104101790 2024-04-24T23:44:39.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.775+0000 7f30ebfff640 1 -- 172.21.15.107:0/2824986204 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f30f409c150 con 0x7f3104101790 2024-04-24T23:44:39.774 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:44:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 msgr2=0x7f30ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 0x7f30ec0a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30f8006010 tx=0x7f30f8000bc0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 msgr2=0x7f3104197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f30f400c970 tx=0x7f30f400ce30 comp rx=0 tx=0).stop 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 shutdown_connections 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f30ec0a2c20 0x7f30ec0a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104103340 0x7f310419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3104102980 0x7f3104197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 --2- 172.21.15.107:0/2824986204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3104101790 0x7f3104197ab0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 >> 172.21.15.107:0/2824986204 conn(0x7f31040fcd20 msgr2=0x7f3104105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 shutdown_connections 2024-04-24T23:44:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.777+0000 7f3109076640 1 -- 172.21.15.107:0/2824986204 wait complete. 2024-04-24T23:44:39.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 -- 172.21.15.107:0/4178841202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f440ffe10 msgr2=0x7f0f44100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 --2- 172.21.15.107:0/4178841202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f440ffe10 0x7f0f44100230 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400aa10 tx=0x7f0f3402b720 comp rx=0 tx=0).stop 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 -- 172.21.15.107:0/4178841202 shutdown_connections 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 --2- 172.21.15.107:0/4178841202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44101920 0x7f0f44108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 --2- 172.21.15.107:0/4178841202 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f44100f60 0x7f0f441013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 --2- 172.21.15.107:0/4178841202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f440ffe10 0x7f0f44100230 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 -- 172.21.15.107:0/4178841202 >> 172.21.15.107:0/4178841202 conn(0x7f0f440fb380 msgr2=0x7f0f440fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 -- 172.21.15.107:0/4178841202 shutdown_connections 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 -- 172.21.15.107:0/4178841202 wait complete. 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.888+0000 7f0f498a5640 1 Processor -- start 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 -- start start 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 0x7f0f44197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44100f60 0x7f0f44198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410b210 con 0x7f0f44101920 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410b090 con 0x7f0f44100f60 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f498a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410b390 con 0x7f0f440ffe10 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46954/0 (socket says 172.21.15.107:46954) 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 -- 172.21.15.107:0/1705633206 learned_addr learned my addr 172.21.15.107:0/1705633206 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f427fc640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44100f60 0x7f0f44198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f42ffd640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 0x7f0f44197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 -- 172.21.15.107:0/1705633206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 msgr2=0x7f0f44197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 0x7f0f44197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 -- 172.21.15.107:0/1705633206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44100f60 msgr2=0x7f0f44198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44100f60 0x7f0f44198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 -- 172.21.15.107:0/1705633206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4419f6f0 con 0x7f0f44101920 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f42ffd640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 0x7f0f44197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.889+0000 7f0f437fe640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00cad0 tx=0x7f0f3c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.890+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00ae20 con 0x7f0f44101920 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.890+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c005020 con 0x7f0f44101920 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.890+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4419f9e0 con 0x7f0f44101920 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.890+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00f6f0 con 0x7f0f44101920 2024-04-24T23:44:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.890+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4419fee0 con 0x7f0f44101920 2024-04-24T23:44:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.891+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f44107190 con 0x7f0f44101920 2024-04-24T23:44:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.892+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0f3c0051d0 con 0x7f0f44101920 2024-04-24T23:44:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.892+0000 7f0f488a3640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 0x7f0f2c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.892+0000 7f0f42ffd640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 0x7f0f2c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.892+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0f3c0cf8c0 con 0x7f0f44101920 2024-04-24T23:44:39.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.893+0000 7f0f42ffd640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 0x7f0f2c0a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000c00 tx=0x7f0f3400a700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:39.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:39.896+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f3c014030 con 0x7f0f44101920 2024-04-24T23:44:40.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.071+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f4402d3c0 con 0x7f0f44101920 2024-04-24T23:44:40.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.071+0000 7f0f488a3640 1 -- 172.21.15.107:0/1705633206 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f0f3c09c360 con 0x7f0f44101920 2024-04-24T23:44:40.071 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 msgr2=0x7f0f2c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 0x7f0f2c0a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000c00 tx=0x7f0f3400a700 comp rx=0 tx=0).stop 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 msgr2=0x7f0f4419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00cad0 tx=0x7f0f3c00cf90 comp rx=0 tx=0).stop 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 shutdown_connections 2024-04-24T23:44:40.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f2c0a2c20 0x7f0f2c0a50e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f44101920 0x7f0f4419f080 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f44100f60 0x7f0f44198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 --2- 172.21.15.107:0/1705633206 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f440ffe10 0x7f0f44197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 >> 172.21.15.107:0/1705633206 conn(0x7f0f440fb380 msgr2=0x7f0f44104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 shutdown_connections 2024-04-24T23:44:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.073+0000 7f0f498a5640 1 -- 172.21.15.107:0/1705633206 wait complete. 2024-04-24T23:44:40.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- 172.21.15.107:0/1170176758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981021f0 msgr2=0x7f5e981045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/1170176758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981021f0 0x7f5e981045e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009ca0 tx=0x7f5e9402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- 172.21.15.107:0/1170176758 shutdown_connections 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/1170176758 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e98107450 0x7f5e981098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/1170176758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98104b20 0x7f5e98106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/1170176758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981021f0 0x7f5e981045e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- 172.21.15.107:0/1170176758 >> 172.21.15.107:0/1170176758 conn(0x7f5e980fbbe0 msgr2=0x7f5e980fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- 172.21.15.107:0/1170176758 shutdown_connections 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- 172.21.15.107:0/1170176758 wait complete. 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 Processor -- start 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- start start 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 0x7f5e98197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 0x7f5e9819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9810d3f0 con 0x7f5e98104b20 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9810d270 con 0x7f5e98107450 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9fd77640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9810d570 con 0x7f5e981021f0 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9d312640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9d312640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46966/0 (socket says 172.21.15.107:46966) 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.187+0000 7f5e9d312640 1 -- 172.21.15.107:0/2119380524 learned_addr learned my addr 172.21.15.107:0/2119380524 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9db13640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 0x7f5e98197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9e314640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 0x7f5e9819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 -- 172.21.15.107:0/2119380524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 msgr2=0x7f5e98197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 0x7f5e98197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 -- 172.21.15.107:0/2119380524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 msgr2=0x7f5e9819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 0x7f5e9819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 -- 172.21.15.107:0/2119380524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9819f6b0 con 0x7f5e98104b20 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9db13640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 0x7f5e98197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9e314640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 0x7f5e9819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9d312640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800d770 tx=0x7f5e8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88017070 con 0x7f5e98104b20 2024-04-24T23:44:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e88009c30 con 0x7f5e98104b20 2024-04-24T23:44:40.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9819f9a0 con 0x7f5e98104b20 2024-04-24T23:44:40.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8800f620 con 0x7f5e98104b20 2024-04-24T23:44:40.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.188+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9819fef0 con 0x7f5e98104b20 2024-04-24T23:44:40.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.189+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e980689b0 con 0x7f5e98104b20 2024-04-24T23:44:40.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.190+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5e8801a070 con 0x7f5e98104b20 2024-04-24T23:44:40.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.191+0000 7f5e8effd640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 0x7f5e840a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.191+0000 7f5e9db13640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 0x7f5e840a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.191+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e880cf0e0 con 0x7f5e98104b20 2024-04-24T23:44:40.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.191+0000 7f5e9db13640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 0x7f5e840a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e94031c10 tx=0x7f5e94009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.194+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e88009de0 con 0x7f5e98104b20 2024-04-24T23:44:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1936809222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1707977438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2824986204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:40 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1705633206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.370+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5e9802d1a0 con 0x7f5e98104b20 2024-04-24T23:44:40.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.371+0000 7f5e8effd640 1 -- 172.21.15.107:0/2119380524 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f5e8809bb80 con 0x7f5e98104b20 2024-04-24T23:44:40.371 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:44:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 msgr2=0x7f5e840a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 0x7f5e840a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e94031c10 tx=0x7f5e94009360 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 msgr2=0x7f5e98197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800d770 tx=0x7f5e8800dc30 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 shutdown_connections 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e840a2c20 0x7f5e840a50e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e98107450 0x7f5e9819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e98104b20 0x7f5e98197fd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 --2- 172.21.15.107:0/2119380524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e981021f0 0x7f5e98197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 >> 172.21.15.107:0/2119380524 conn(0x7f5e980fbbe0 msgr2=0x7f5e98102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 shutdown_connections 2024-04-24T23:44:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.373+0000 7f5e9fd77640 1 -- 172.21.15.107:0/2119380524 wait complete. 2024-04-24T23:44:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1936809222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1707977438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2824986204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:40 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1705633206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:44:40.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 -- 172.21.15.107:0/904264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c101cc0 msgr2=0x7faf5c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 --2- 172.21.15.107:0/904264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c101cc0 0x7faf5c1020e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009ca0 tx=0x7faf4c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 -- 172.21.15.107:0/904264621 shutdown_connections 2024-04-24T23:44:40.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 --2- 172.21.15.107:0/904264621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c1013b0 0x7faf5c108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 --2- 172.21.15.107:0/904264621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 --2- 172.21.15.107:0/904264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c101cc0 0x7faf5c1020e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 -- 172.21.15.107:0/904264621 >> 172.21.15.107:0/904264621 conn(0x7faf5c0fb420 msgr2=0x7faf5c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 -- 172.21.15.107:0/904264621 shutdown_connections 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.480+0000 7faf6314f640 1 -- 172.21.15.107:0/904264621 wait complete. 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 Processor -- start 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 -- start start 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c10b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 0x7faf5c109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c10d520 con 0x7faf5c1013b0 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c10d3a0 con 0x7faf5c1009f0 2024-04-24T23:44:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf6314f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c10d6a0 con 0x7faf5c101cc0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46988/0 (socket says 172.21.15.107:46988) 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 -- 172.21.15.107:0/3841985618 learned_addr learned my addr 172.21.15.107:0/3841985618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf616ec640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 0x7faf5c109c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 -- 172.21.15.107:0/3841985618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 msgr2=0x7faf5c109c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf60eeb640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c10b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 0x7faf5c109c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 -- 172.21.15.107:0/3841985618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 msgr2=0x7faf5c10b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c10b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf5bfff640 1 -- 172.21.15.107:0/3841985618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c10a4a0 con 0x7faf5c1013b0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf616ec640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 0x7faf5c109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.481+0000 7faf60eeb640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c10b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf5bfff640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c890 tx=0x7faf5000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50004220 con 0x7faf5c1013b0 2024-04-24T23:44:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c1a1ce0 con 0x7faf5c1013b0 2024-04-24T23:44:40.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf500043c0 con 0x7faf5c1013b0 2024-04-24T23:44:40.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500049e0 con 0x7faf5c1013b0 2024-04-24T23:44:40.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.482+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c1a2180 con 0x7faf5c1013b0 2024-04-24T23:44:40.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.483+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5c107b30 con 0x7faf5c1013b0 2024-04-24T23:44:40.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.484+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7faf500079b0 con 0x7faf5c1013b0 2024-04-24T23:44:40.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.484+0000 7faf59ffb640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 0x7faf440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.485+0000 7faf60eeb640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 0x7faf440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.485+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faf500cf6d0 con 0x7faf5c1013b0 2024-04-24T23:44:40.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.485+0000 7faf60eeb640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 0x7faf440a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.489+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf50014030 con 0x7faf5c1013b0 2024-04-24T23:44:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1936809222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1707977438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2824986204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:40 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1705633206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:40.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.705+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf5c02d370 con 0x7faf5c1013b0 2024-04-24T23:44:40.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.706+0000 7faf59ffb640 1 -- 172.21.15.107:0/3841985618 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7faf5009c170 con 0x7faf5c1013b0 2024-04-24T23:44:40.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:40.707 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:44:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 msgr2=0x7faf440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 0x7faf440a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c000f40 comp rx=0 tx=0).stop 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 msgr2=0x7faf5c1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c890 tx=0x7faf5000cd50 comp rx=0 tx=0).stop 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 shutdown_connections 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf440a2c20 0x7faf440a50e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf5c101cc0 0x7faf5c109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf5c1013b0 0x7faf5c1096d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 --2- 172.21.15.107:0/3841985618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf5c1009f0 0x7faf5c10b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 >> 172.21.15.107:0/3841985618 conn(0x7faf5c0fb420 msgr2=0x7faf5c104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 shutdown_connections 2024-04-24T23:44:40.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.708+0000 7faf6314f640 1 -- 172.21.15.107:0/3841985618 wait complete. 2024-04-24T23:44:40.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:44:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 -- 172.21.15.107:0/2246589639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864104b50 msgr2=0x7fb864106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 --2- 172.21.15.107:0/2246589639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864104b50 0x7fb864106f40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb858009900 tx=0x7fb85802a410 comp rx=0 tx=0).stop 2024-04-24T23:44:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 -- 172.21.15.107:0/2246589639 shutdown_connections 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 --2- 172.21.15.107:0/2246589639 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 0x7fb8641098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 --2- 172.21.15.107:0/2246589639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864104b50 0x7fb864106f40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 --2- 172.21.15.107:0/2246589639 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb864102220 0x7fb864104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 -- 172.21.15.107:0/2246589639 >> 172.21.15.107:0/2246589639 conn(0x7fb8640fbc30 msgr2=0x7fb8640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 -- 172.21.15.107:0/2246589639 shutdown_connections 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.817+0000 7fb869db6640 1 -- 172.21.15.107:0/2246589639 wait complete. 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 Processor -- start 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 -- start start 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb864104b50 0x7fb864198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 0x7fb86419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410d430 con 0x7fb864102220 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410d2b0 con 0x7fb864107480 2024-04-24T23:44:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb869db6640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410d5b0 con 0x7fb864104b50 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47014/0 (socket says 172.21.15.107:47014) 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 -- 172.21.15.107:0/1562039089 learned_addr learned my addr 172.21.15.107:0/1562039089 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 -- 172.21.15.107:0/1562039089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb864104b50 msgr2=0x7fb864198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb864104b50 0x7fb864198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb863fff640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 0x7fb86419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 -- 172.21.15.107:0/1562039089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 msgr2=0x7fb86419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.818+0000 7fb8637fe640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 0x7fb86419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb8637fe640 1 -- 172.21.15.107:0/1562039089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86419f720 con 0x7fb864102220 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb8637fe640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb85400e9f0 tx=0x7fb85400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85400cd80 con 0x7fb864102220 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86419fa10 con 0x7fb864102220 2024-04-24T23:44:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb854005500 con 0x7fb864102220 2024-04-24T23:44:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854010690 con 0x7fb864102220 2024-04-24T23:44:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.819+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb86419ff60 con 0x7fb864102220 2024-04-24T23:44:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.820+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8640fd720 con 0x7fb864102220 2024-04-24T23:44:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.821+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb8540046c0 con 0x7fb864102220 2024-04-24T23:44:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.821+0000 7fb860ff9640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 0x7fb84c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.822+0000 7fb862ffd640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 0x7fb84c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.822+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8540cfc90 con 0x7fb864102220 2024-04-24T23:44:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.822+0000 7fb862ffd640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 0x7fb84c0a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb8580096c0 tx=0x7fb858031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:40.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:40.825+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb854053740 con 0x7fb864102220 2024-04-24T23:44:41.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.003+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb864062340 con 0x7fb864102220 2024-04-24T23:44:41.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.005+0000 7fb860ff9640 1 -- 172.21.15.107:0/1562039089 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24715 (secure 0 0 0) 0x7fb85409c730 con 0x7fb864102220 2024-04-24T23:44:41.006 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:41.006 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:43:49.212670+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:44:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 msgr2=0x7fb84c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 0x7fb84c0a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb8580096c0 tx=0x7fb858031400 comp rx=0 tx=0).stop 2024-04-24T23:44:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 msgr2=0x7fb864197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb85400e9f0 tx=0x7fb85400edf0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 shutdown_connections 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb84c0a2c20 0x7fb84c0a50e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb864107480 0x7fb86419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb864104b50 0x7fb864198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 --2- 172.21.15.107:0/1562039089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb864102220 0x7fb864197b00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 >> 172.21.15.107:0/1562039089 conn(0x7fb8640fbc30 msgr2=0x7fb864102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 shutdown_connections 2024-04-24T23:44:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.007+0000 7fb869db6640 1 -- 172.21.15.107:0/1562039089 wait complete. 2024-04-24T23:44:41.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:44:41.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:44:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 -- 172.21.15.107:0/1430685045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8102230 msgr2=0x7f57c8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 --2- 172.21.15.107:0/1430685045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8102230 0x7f57c8108ac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009900 tx=0x7f57c002a410 comp rx=0 tx=0).stop 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 -- 172.21.15.107:0/1430685045 shutdown_connections 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 --2- 172.21.15.107:0/1430685045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8102230 0x7f57c8108ac0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 --2- 172.21.15.107:0/1430685045 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8101870 0x7f57c8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 --2- 172.21.15.107:0/1430685045 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8100680 0x7f57c8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 -- 172.21.15.107:0/1430685045 >> 172.21.15.107:0/1430685045 conn(0x7f57c80fbbf0 msgr2=0x7f57c80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 -- 172.21.15.107:0/1430685045 shutdown_connections 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.120+0000 7f57cf929640 1 -- 172.21.15.107:0/1430685045 wait complete. 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 Processor -- start 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 -- start start 2024-04-24T23:44:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 0x7f57c819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8102230 0x7f57c819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c2e0 con 0x7f57c8100680 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c160 con 0x7f57c8101870 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cf929640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c460 con 0x7f57c8102230 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47024/0 (socket says 172.21.15.107:47024) 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 -- 172.21.15.107:0/161452265 learned_addr learned my addr 172.21.15.107:0/161452265 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57ccec4640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 0x7f57c819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cdec6640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8102230 0x7f57c819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 -- 172.21.15.107:0/161452265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8102230 msgr2=0x7f57c819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8102230 0x7f57c819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 -- 172.21.15.107:0/161452265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 msgr2=0x7f57c819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 0x7f57c819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 -- 172.21.15.107:0/161452265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c819f670 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57ccec4640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 0x7f57c819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.121+0000 7f57cd6c5640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f57b800ccb0 tx=0x7f57b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.122+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8003990 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.122+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c819f960 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.122+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57b8003b30 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.122+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b800f7b0 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.122+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c819feb0 con 0x7f57c8100680 2024-04-24T23:44:41.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.123+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c8107940 con 0x7f57c8100680 2024-04-24T23:44:41.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.124+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f57b800f950 con 0x7f57c8100680 2024-04-24T23:44:41.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.124+0000 7f57c67fc640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 0x7f57b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.125+0000 7f57ccec4640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 0x7f57b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.125+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f57b80cf720 con 0x7f57c8100680 2024-04-24T23:44:41.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.125+0000 7f57ccec4640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 0x7f57b00a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57c80691d0 tx=0x7f57bc00a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.129+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57b8014030 con 0x7f57c8100680 2024-04-24T23:44:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.305+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f57c802d170 con 0x7f57c8100680 2024-04-24T23:44:41.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.305+0000 7f57c67fc640 1 -- 172.21.15.107:0/161452265 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f57b809c1c0 con 0x7f57c8100680 2024-04-24T23:44:41.306 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:44:41.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 msgr2=0x7f57b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 0x7f57b00a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57c80691d0 tx=0x7f57bc00a5c0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 msgr2=0x7f57c8069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f57b800ccb0 tx=0x7f57b800a620 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 shutdown_connections 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f57b00a2c20 0x7f57b00a50e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f57c8102230 0x7f57c819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57c8101870 0x7f57c819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 --2- 172.21.15.107:0/161452265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57c8100680 0x7f57c8069460 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 >> 172.21.15.107:0/161452265 conn(0x7f57c80fbbf0 msgr2=0x7f57c8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.307+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 shutdown_connections 2024-04-24T23:44:41.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.308+0000 7f57cf929640 1 -- 172.21.15.107:0/161452265 wait complete. 2024-04-24T23:44:41.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:41 smithi107 ceph-mon[29118]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:41.318 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:41 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2119380524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:41.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:41 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3841985618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:41 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1562039089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:44:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:41 smithi170 ceph-mon[32714]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:41 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2119380524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:41 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3841985618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:41 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1562039089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1682625316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30107500 msgr2=0x7ffa301098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1682625316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30107500 0x7ffa301098f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009ca0 tx=0x7ffa2802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1682625316 shutdown_connections 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1682625316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30107500 0x7ffa301098f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1682625316 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa30104bd0 0x7ffa30106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1682625316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa301022a0 0x7ffa30104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1682625316 >> 172.21.15.107:0/1682625316 conn(0x7ffa300fbc10 msgr2=0x7ffa300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1682625316 shutdown_connections 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.419+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1682625316 wait complete. 2024-04-24T23:44:41.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 Processor -- start 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 -- start start 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa301022a0 0x7ffa30101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa30107500 0x7ffa30100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010d440 con 0x7ffa30104bd0 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010d2c0 con 0x7ffa301022a0 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa35ebd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010d5c0 con 0x7ffa30107500 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47040/0 (socket says 172.21.15.107:47040) 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 -- 172.21.15.107:0/1590130786 learned_addr learned my addr 172.21.15.107:0/1590130786 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 -- 172.21.15.107:0/1590130786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa30107500 msgr2=0x7ffa30100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa30107500 0x7ffa30100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 -- 172.21.15.107:0/1590130786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa301022a0 msgr2=0x7ffa30101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa301022a0 0x7ffa30101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 -- 172.21.15.107:0/1590130786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa30101020 con 0x7ffa30104bd0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.420+0000 7ffa2effd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400e9f0 tx=0x7ffa2400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.421+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400ccc0 con 0x7ffa30104bd0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.421+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa24005500 con 0x7ffa30104bd0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.421+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa301a1bd0 con 0x7ffa30104bd0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.421+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24010690 con 0x7ffa30104bd0 2024-04-24T23:44:41.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.421+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa301a20d0 con 0x7ffa30104bd0 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.422+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa300fd700 con 0x7ffa30104bd0 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.423+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ffa240046c0 con 0x7ffa30104bd0 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.423+0000 7ffa2cff9640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 0x7ffa180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.423+0000 7ffa2f7fe640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 0x7ffa180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.423+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ffa240cfa40 con 0x7ffa30104bd0 2024-04-24T23:44:41.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.424+0000 7ffa2f7fe640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 0x7ffa180a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa200058c0 tx=0x7ffa20005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.428+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa2409c4e0 con 0x7ffa30104bd0 2024-04-24T23:44:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:41 smithi195 ceph-mon[33019]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:41 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2119380524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:41 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3841985618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:41 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1562039089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:41.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.603+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa3002d360 con 0x7ffa30104bd0 2024-04-24T23:44:41.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.603+0000 7ffa2cff9640 1 -- 172.21.15.107:0/1590130786 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7ffa2409bde0 con 0x7ffa30104bd0 2024-04-24T23:44:41.604 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:44:41.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 msgr2=0x7ffa180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 0x7ffa180a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa200058c0 tx=0x7ffa20005850 comp rx=0 tx=0).stop 2024-04-24T23:44:41.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 msgr2=0x7ffa30100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400e9f0 tx=0x7ffa2400eeb0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 shutdown_connections 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa180a2c20 0x7ffa180a50e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa30107500 0x7ffa30100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30104bd0 0x7ffa30100250 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 --2- 172.21.15.107:0/1590130786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa301022a0 0x7ffa30101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 >> 172.21.15.107:0/1590130786 conn(0x7ffa300fbc10 msgr2=0x7ffa30102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 shutdown_connections 2024-04-24T23:44:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.605+0000 7ffa35ebd640 1 -- 172.21.15.107:0/1590130786 wait complete. 2024-04-24T23:44:41.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:44:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3113417406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 msgr2=0x7fc254108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3113417406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc254108ae0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc250009ca0 tx=0x7fc25002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3113417406 shutdown_connections 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3113417406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc254108ae0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3113417406 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc254101890 0x7fc254101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3113417406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2541006a0 0x7fc254100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3113417406 >> 172.21.15.107:0/3113417406 conn(0x7fc2540fbc30 msgr2=0x7fc2540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3113417406 shutdown_connections 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.717+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3113417406 wait complete. 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 Processor -- start 2024-04-24T23:44:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 -- start start 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2541006a0 0x7fc254196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc254101890 0x7fc254196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410c300 con 0x7fc254102250 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410c180 con 0x7fc254101890 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25c2d4640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410c480 con 0x7fc2541006a0 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47062/0 (socket says 172.21.15.107:47062) 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 -- 172.21.15.107:0/3456976372 learned_addr learned my addr 172.21.15.107:0/3456976372 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25986f640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc254101890 0x7fc254196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a070640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2541006a0 0x7fc254196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 -- 172.21.15.107:0/3456976372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2541006a0 msgr2=0x7fc254196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2541006a0 0x7fc254196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 -- 172.21.15.107:0/3456976372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc254101890 msgr2=0x7fc254196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc254101890 0x7fc254196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.718+0000 7fc25a871640 1 -- 172.21.15.107:0/3456976372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25419e440 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc25a871640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2500098d0 tx=0x7fc25002f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25002fc30 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc250003e50 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25419e6d0 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250004810 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.719+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25419ec20 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.720+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2541078f0 con 0x7fc254102250 2024-04-24T23:44:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.721+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc250003970 con 0x7fc254102250 2024-04-24T23:44:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.721+0000 7fc24b7fe640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 0x7fc2400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.721+0000 7fc25a070640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 0x7fc2400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.721+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc2500f0770 con 0x7fc254102250 2024-04-24T23:44:41.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.722+0000 7fc25a070640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 0x7fc2400a50e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2541016f0 tx=0x7fc24c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:41.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.726+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc2500bd440 con 0x7fc254102250 2024-04-24T23:44:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.901+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc254062340 con 0x7fc254102250 2024-04-24T23:44:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.901+0000 7fc24b7fe640 1 -- 172.21.15.107:0/3456976372 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fc2500bcd40 con 0x7fc254102250 2024-04-24T23:44:41.902 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:44:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 msgr2=0x7fc2400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 0x7fc2400a50e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc2541016f0 tx=0x7fc24c005f90 comp rx=0 tx=0).stop 2024-04-24T23:44:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 msgr2=0x7fc25419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2500098d0 tx=0x7fc25002f840 comp rx=0 tx=0).stop 2024-04-24T23:44:41.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 shutdown_connections 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc2400a2c20 0x7fc2400a50e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc254102250 0x7fc25419ddd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc254101890 0x7fc254196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 --2- 172.21.15.107:0/3456976372 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc2541006a0 0x7fc254196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 >> 172.21.15.107:0/3456976372 conn(0x7fc2540fbc30 msgr2=0x7fc254104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 shutdown_connections 2024-04-24T23:44:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:41.903+0000 7fc25c2d4640 1 -- 172.21.15.107:0/3456976372 wait complete. 2024-04-24T23:44:41.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:44:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 -- 172.21.15.107:0/896210881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 msgr2=0x7f08f01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/896210881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01098b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009ca0 tx=0x7f08e802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 -- 172.21.15.107:0/896210881 shutdown_connections 2024-04-24T23:44:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/896210881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01098b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/896210881 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0104b30 0x7f08f0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/896210881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0102200 0x7f08f01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 -- 172.21.15.107:0/896210881 >> 172.21.15.107:0/896210881 conn(0x7f08f00fbc30 msgr2=0x7f08f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 -- 172.21.15.107:0/896210881 shutdown_connections 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.016+0000 7f08f5dd5640 1 -- 172.21.15.107:0/896210881 wait complete. 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 Processor -- start 2024-04-24T23:44:42.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 -- start start 2024-04-24T23:44:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0102200 0x7f08f0101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0104b30 0x7f08f01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08f010d420 con 0x7f08f0107460 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f08f010d2a0 con 0x7f08f0104b30 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08f5dd5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f08f010d5a0 con 0x7f08f0102200 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47086/0 (socket says 172.21.15.107:47086) 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 -- 172.21.15.107:0/2632565029 learned_addr learned my addr 172.21.15.107:0/2632565029 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08ef7fe640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0102200 0x7f08f0101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08eeffd640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0104b30 0x7f08f01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 -- 172.21.15.107:0/2632565029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0102200 msgr2=0x7f08f0101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0102200 0x7f08f0101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 -- 172.21.15.107:0/2632565029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0104b30 msgr2=0x7f08f01001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0104b30 0x7f08f01001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.017+0000 7f08effff640 1 -- 172.21.15.107:0/2632565029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f0100f80 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08effff640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005e30 tx=0x7f08e8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8003c30 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e8003dd0 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f01a1b80 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e802fde0 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.018+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f01a20d0 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.019+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f00fd720 con 0x7f08f0107460 2024-04-24T23:44:42.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.020+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f08e8004050 con 0x7f08f0107460 2024-04-24T23:44:42.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.021+0000 7f08ecff9640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 0x7f08d80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.021+0000 7f08ef7fe640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 0x7f08d80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.021+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f08e80f0a30 con 0x7f08f0107460 2024-04-24T23:44:42.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.021+0000 7f08ef7fe640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 0x7f08d80a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08e00058c0 tx=0x7f08e000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.024+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08e80bd580 con 0x7f08f0107460 2024-04-24T23:44:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.206+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f08f0062340 con 0x7f08f0107460 2024-04-24T23:44:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.206+0000 7f08ecff9640 1 -- 172.21.15.107:0/2632565029 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f08e80bce80 con 0x7f08f0107460 2024-04-24T23:44:42.207 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 msgr2=0x7f08d80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 0x7f08d80a50e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08e00058c0 tx=0x7f08e000a2d0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 msgr2=0x7f08f01006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005e30 tx=0x7f08e8031ee0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 shutdown_connections 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f08d80a2c20 0x7f08d80a50e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f0107460 0x7f08f01006f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f0104b30 0x7f08f01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 --2- 172.21.15.107:0/2632565029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f08f0102200 0x7f08f0101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 >> 172.21.15.107:0/2632565029 conn(0x7f08f00fbc30 msgr2=0x7f08f0102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 shutdown_connections 2024-04-24T23:44:42.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.208+0000 7f08f5dd5640 1 -- 172.21.15.107:0/2632565029 wait complete. 2024-04-24T23:44:42.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:42 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/161452265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:42 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1590130786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:42 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3456976372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:44:42.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 -- 172.21.15.107:0/2092662770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798100680 msgr2=0x7f3798100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 --2- 172.21.15.107:0/2092662770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798100680 0x7f3798100aa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3794009900 tx=0x7f379402a410 comp rx=0 tx=0).stop 2024-04-24T23:44:42.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 -- 172.21.15.107:0/2092662770 shutdown_connections 2024-04-24T23:44:42.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 --2- 172.21.15.107:0/2092662770 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f3798108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 --2- 172.21.15.107:0/2092662770 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798101870 0x7f3798101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 --2- 172.21.15.107:0/2092662770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798100680 0x7f3798100aa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 -- 172.21.15.107:0/2092662770 >> 172.21.15.107:0/2092662770 conn(0x7f37980fbbf0 msgr2=0x7f37980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 -- 172.21.15.107:0/2092662770 shutdown_connections 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.318+0000 7f37a01b0640 1 -- 172.21.15.107:0/2092662770 wait complete. 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 Processor -- start 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 -- start start 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 0x7f3798069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f379819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f379810c350 con 0x7f3798101870 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f379810c1d0 con 0x7f3798102230 2024-04-24T23:44:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f37a01b0640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f379810c4d0 con 0x7f3798100680 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47114/0 (socket says 172.21.15.107:47114) 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 -- 172.21.15.107:0/3460872301 learned_addr learned my addr 172.21.15.107:0/3460872301 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379df4c640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 0x7f3798069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379e74d640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f379819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 -- 172.21.15.107:0/3460872301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 msgr2=0x7f3798069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 0x7f3798069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 -- 172.21.15.107:0/3460872301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 msgr2=0x7f379819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f379819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 -- 172.21.15.107:0/3460872301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379819f720 con 0x7f3798101870 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379df4c640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 0x7f3798069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379e74d640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f379819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f379d74b640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f378800c920 tx=0x7f378800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.319+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788007c70 con 0x7f3798101870 2024-04-24T23:44:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.320+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788007e10 con 0x7f3798101870 2024-04-24T23:44:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.320+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f379819f9b0 con 0x7f3798101870 2024-04-24T23:44:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.320+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788018890 con 0x7f3798101870 2024-04-24T23:44:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.320+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f379819ffd0 con 0x7f3798101870 2024-04-24T23:44:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.321+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3798107940 con 0x7f3798101870 2024-04-24T23:44:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.322+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f3788018a70 con 0x7f3798101870 2024-04-24T23:44:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.322+0000 7f378effd640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 0x7f37840a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.322+0000 7f379df4c640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 0x7f37840a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.322+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37880cf690 con 0x7f3798101870 2024-04-24T23:44:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.323+0000 7f379df4c640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 0x7f37840a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3794000c00 tx=0x7f3794031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.327+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3788014030 con 0x7f3798101870 2024-04-24T23:44:42.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.538+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f379802d170 con 0x7f3798101870 2024-04-24T23:44:42.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.539+0000 7f378effd640 1 -- 172.21.15.107:0/3460872301 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7f378809c130 con 0x7f3798101870 2024-04-24T23:44:42.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:42.540 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:44:42.540 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:44:42.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 msgr2=0x7f37840a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 0x7f37840a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3794000c00 tx=0x7f3794031400 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 msgr2=0x7f379819c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f378800c920 tx=0x7f378800cde0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 shutdown_connections 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f37840a2ca0 0x7f37840a5160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3798102230 0x7f379819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3798101870 0x7f379819c7f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 --2- 172.21.15.107:0/3460872301 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3798100680 0x7f3798069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 >> 172.21.15.107:0/3460872301 conn(0x7f37980fbbf0 msgr2=0x7f3798104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 shutdown_connections 2024-04-24T23:44:42.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.541+0000 7f37a01b0640 1 -- 172.21.15.107:0/3460872301 wait complete. 2024-04-24T23:44:42.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:44:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:42 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/161452265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:42 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1590130786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:42 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3456976372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:42 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/161452265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:42 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1590130786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:42 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3456976372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:44:42.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 -- 172.21.15.107:0/1796875628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 msgr2=0x7f5034100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 --2- 172.21.15.107:0/1796875628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f5034100ab0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5024009900 tx=0x7f502402a410 comp rx=0 tx=0).stop 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 -- 172.21.15.107:0/1796875628 shutdown_connections 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 --2- 172.21.15.107:0/1796875628 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034102240 0x7f5034108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 --2- 172.21.15.107:0/1796875628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5034101880 0x7f5034101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 --2- 172.21.15.107:0/1796875628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f5034100ab0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 -- 172.21.15.107:0/1796875628 >> 172.21.15.107:0/1796875628 conn(0x7f50340fbc20 msgr2=0x7f50340fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 -- 172.21.15.107:0/1796875628 shutdown_connections 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.650+0000 7f5039b12640 1 -- 172.21.15.107:0/1796875628 wait complete. 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 Processor -- start 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 -- start start 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 0x7f503419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5034102240 0x7f503419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c360 con 0x7f5034100690 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c1e0 con 0x7f5034102240 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5039b12640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c4e0 con 0x7f5034101880 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f50337fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f50337fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47142/0 (socket says 172.21.15.107:47142) 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f50337fe640 1 -- 172.21.15.107:0/3376975388 learned_addr learned my addr 172.21.15.107:0/3376975388 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.651+0000 7f5032ffd640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 0x7f503419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 -- 172.21.15.107:0/3376975388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 msgr2=0x7f503419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 0x7f503419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 -- 172.21.15.107:0/3376975388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5034102240 msgr2=0x7f503419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5034102240 0x7f503419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 -- 172.21.15.107:0/3376975388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503419f760 con 0x7f5034100690 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5032ffd640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 0x7f503419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:42.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f50337fe640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5024031610 tx=0x7f50240043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502403f030 con 0x7f5034100690 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503419f9f0 con 0x7f5034100690 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5024003ce0 con 0x7f5034100690 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f503419ffb0 con 0x7f5034100690 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.652+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024049060 con 0x7f5034100690 2024-04-24T23:44:42.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.653+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5034107a00 con 0x7f5034100690 2024-04-24T23:44:42.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.654+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5024042030 con 0x7f5034100690 2024-04-24T23:44:42.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.655+0000 7f5030ff9640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 0x7f50200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.655+0000 7f5032ffd640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 0x7f50200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.655+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f50240f02c0 con 0x7f5034100690 2024-04-24T23:44:42.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.655+0000 7f5032ffd640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 0x7f50200a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f503406a6a0 tx=0x7f501c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.658+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f50240bcd60 con 0x7f5034100690 2024-04-24T23:44:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.833+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f503402d3e0 con 0x7f5034100690 2024-04-24T23:44:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.835+0000 7f5030ff9640 1 -- 172.21.15.107:0/3376975388 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24715 (secure 0 0 0) 0x7f50240bc660 con 0x7f5034100690 2024-04-24T23:44:42.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:42.836 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:43:49.212670+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 msgr2=0x7f50200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 0x7f50200a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f503406a6a0 tx=0x7f501c005d30 comp rx=0 tx=0).stop 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 msgr2=0x7f503406a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5024031610 tx=0x7f50240043b0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 shutdown_connections 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50200a2c20 0x7f50200a50e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5034102240 0x7f503419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5034101880 0x7f503419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 --2- 172.21.15.107:0/3376975388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5034100690 0x7f503406a930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 >> 172.21.15.107:0/3376975388 conn(0x7f50340fbc20 msgr2=0x7f5034104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 shutdown_connections 2024-04-24T23:44:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.837+0000 7f5039b12640 1 -- 172.21.15.107:0/3376975388 wait complete. 2024-04-24T23:44:42.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:44:42.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 -- 172.21.15.107:0/3904383936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 msgr2=0x7f1e50106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 --2- 172.21.15.107:0/3904383936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50106fe0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009900 tx=0x7f1e4c02a410 comp rx=0 tx=0).stop 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 -- 172.21.15.107:0/3904383936 shutdown_connections 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 --2- 172.21.15.107:0/3904383936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e50109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 --2- 172.21.15.107:0/3904383936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50106fe0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 --2- 172.21.15.107:0/3904383936 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 0x7f1e501046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.947+0000 7f1e58093640 1 -- 172.21.15.107:0/3904383936 >> 172.21.15.107:0/3904383936 conn(0x7f1e500fbbf0 msgr2=0x7f1e500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- 172.21.15.107:0/3904383936 shutdown_connections 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- 172.21.15.107:0/3904383936 wait complete. 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 Processor -- start 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- start start 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 0x7f1e50197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e5019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5010d400 con 0x7f1e50104bf0 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5010d280 con 0x7f1e50107520 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e58093640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5010d580 con 0x7f1e501022c0 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e5562e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e5562e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47156/0 (socket says 172.21.15.107:47156) 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e5562e640 1 -- 172.21.15.107:0/1813994124 learned_addr learned my addr 172.21.15.107:0/1813994124 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.948+0000 7f1e55e2f640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 0x7f1e50197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e56630640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e5019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 -- 172.21.15.107:0/1813994124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 msgr2=0x7f1e50197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 0x7f1e50197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 -- 172.21.15.107:0/1813994124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 msgr2=0x7f1e5019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e5019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:42.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 -- 172.21.15.107:0/1813994124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5019f720 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e56630640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e5019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e5562e640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c031610 tx=0x7f1e4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c03f030 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5019f9b0 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e4c003ad0 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c035070 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.949+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5019fea0 con 0x7f1e50104bf0 2024-04-24T23:44:42.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.950+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e500fd6e0 con 0x7f1e50104bf0 2024-04-24T23:44:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.951+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1e4c042030 con 0x7f1e50104bf0 2024-04-24T23:44:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.951+0000 7f1e42ffd640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 0x7f1e3c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.952+0000 7f1e55e2f640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 0x7f1e3c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.952+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e4c0efd10 con 0x7f1e50104bf0 2024-04-24T23:44:42.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.952+0000 7f1e55e2f640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 0x7f1e3c0a50e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009b60 tx=0x7f1e44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:42.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:42.956+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e4c0bc7b0 con 0x7f1e50104bf0 2024-04-24T23:44:43.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:43 smithi107 ceph-mon[29118]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:43.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:43 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2632565029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:43 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3460872301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:43 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3376975388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.175+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e5002d3c0 con 0x7f1e50104bf0 2024-04-24T23:44:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.175+0000 7f1e42ffd640 1 -- 172.21.15.107:0/1813994124 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7f1e4c0bc0b0 con 0x7f1e50104bf0 2024-04-24T23:44:43.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:44:43.176 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:44:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:44:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 msgr2=0x7f1e3c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 0x7f1e3c0a50e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009b60 tx=0x7f1e44005f90 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 msgr2=0x7f1e50198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c031610 tx=0x7f1e4c004270 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 shutdown_connections 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e3c0a2c20 0x7f1e3c0a50e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e50107520 0x7f1e5019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50104bf0 0x7f1e50198040 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 --2- 172.21.15.107:0/1813994124 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e501022c0 0x7f1e50197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.177+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 >> 172.21.15.107:0/1813994124 conn(0x7f1e500fbbf0 msgr2=0x7f1e50102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.178+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 shutdown_connections 2024-04-24T23:44:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.178+0000 7f1e58093640 1 -- 172.21.15.107:0/1813994124 wait complete. 2024-04-24T23:44:43.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T23:42:50.405146+0000', 'modified': '2024-04-24T23:43:19.093489+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24493, 'mds_2': 24464, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6839/527084969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 527084969}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 527084969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6841/1612113576', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6840', 'nonce': 1612113576}, {'type': 'v1', 'addr': '172.21.15.107:6841', 'nonce': 1612113576}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24406': {'gid': 24406, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.195:6833/1442081594', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1442081594}, {'type': 'v1', 'addr': '172.21.15.195:6833', 'nonce': 1442081594}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6833/3444431424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3444431424}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 3444431424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/3179503985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 3179503985}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3179503985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.195:6835/1997811297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6834', 'nonce': 1997811297}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 1997811297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.195:6837/1315131786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6836', 'nonce': 1315131786}, {'type': 'v1', 'addr': '172.21.15.195:6837', 'nonce': 1315131786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.170:6837/293399389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 293399389}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 293399389}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.195:6839/4197811615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6838', 'nonce': 4197811615}, {'type': 'v1', 'addr': '172.21.15.195:6839', 'nonce': 4197811615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6839/2021576737', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 2021576737}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 2021576737}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24464, 24506, 24463, 24493, 24451]} 2024-04-24T23:44:43.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-24T23:44:43.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 -- 172.21.15.107:0/371278895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc109640 msgr2=0x7fd3cc10bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 --2- 172.21.15.107:0/371278895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc109640 0x7fd3cc10bad0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009ca0 tx=0x7fd3c402a950 comp rx=0 tx=0).stop 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 -- 172.21.15.107:0/371278895 shutdown_connections 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 --2- 172.21.15.107:0/371278895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc109640 0x7fd3cc10bad0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 --2- 172.21.15.107:0/371278895 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3cc074120 0x7fd3cc074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 --2- 172.21.15.107:0/371278895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3cc075700 0x7fd3cc073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 -- 172.21.15.107:0/371278895 >> 172.21.15.107:0/371278895 conn(0x7fd3cc0fcd20 msgr2=0x7fd3cc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 -- 172.21.15.107:0/371278895 shutdown_connections 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.288+0000 7fd3d2038640 1 -- 172.21.15.107:0/371278895 wait complete. 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 Processor -- start 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 -- start start 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3cc075700 0x7fd3cc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3cc109640 0x7fd3cc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10f600 con 0x7fd3cc074120 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10f480 con 0x7fd3cc109640 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3d2038640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10f780 con 0x7fd3cc075700 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47170/0 (socket says 172.21.15.107:47170) 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3436599058 learned_addr learned my addr 172.21.15.107:0/3436599058 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3436599058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3cc075700 msgr2=0x7fd3cc19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3cc075700 0x7fd3cc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3436599058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3cc109640 msgr2=0x7fd3cc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3cc109640 0x7fd3cc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.289+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3436599058 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc1a1800 con 0x7fd3cc074120 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00d950 tx=0x7fd3bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3c97fa640 1 -- 172.21.15.107:0/3436599058 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc017030 con 0x7fd3cc074120 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3c97fa640 1 -- 172.21.15.107:0/3436599058 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc004020 con 0x7fd3cc074120 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc1a1af0 con 0x7fd3cc074120 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3c97fa640 1 -- 172.21.15.107:0/3436599058 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0047a0 con 0x7fd3cc074120 2024-04-24T23:44:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.290+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc1a2040 con 0x7fd3cc074120 2024-04-24T23:44:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.292+0000 7fd3c97fa640 1 -- 172.21.15.107:0/3436599058 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd3bc0041d0 con 0x7fd3cc074120 2024-04-24T23:44:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.292+0000 7fd3d4e29740 1 Processor -- start 2024-04-24T23:44:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.292+0000 7fd3d4e29740 1 -- start start 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x56467454f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674557430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646744b2f00 con 0x56467454fe30 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56467445b750 con 0x564674554fa0 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5646744aeec0 con 0x56467454f440 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3c97fa640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 0x7fd3b80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47176/0 (socket says 172.21.15.107:47176) 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/2951757520 learned_addr learned my addr 172.21.15.107:0/2951757520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674557430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 0x7fd3b80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x56467454f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3c97fa640 1 -- 172.21.15.107:0/3436599058 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3bc0cefe0 con 0x7fd3cc074120 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/2951757520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 msgr2=0x56467454f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x56467454f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/2951757520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 msgr2=0x564674557430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674557430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/2951757520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646741c72d0 con 0x56467454fe30 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09b060 tx=0x7fd3bc00eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7dcfa631a3ae57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 0x7fd3b80a50e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x564674550690 tx=0x7fd3c402f940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.293+0000 7fd3c1ffb640 1 -- 172.21.15.107:0/2951757520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc09ba60 con 0x56467454fe30 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3c1ffb640 1 -- 172.21.15.107:0/2951757520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc09b360 con 0x56467454fe30 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3c1ffb640 1 -- 172.21.15.107:0/2951757520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0a3b40 con 0x56467454fe30 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2951757520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 msgr2=0x564674554930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09b060 tx=0x7fd3bc00eed0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2951757520 shutdown_connections 2024-04-24T23:44:43.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674557430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674554930 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2951757520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x56467454f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2951757520 >> 172.21.15.107:0/2951757520 conn(0x56467454aab0 msgr2=0x56467454aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2951757520 shutdown_connections 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2951757520 wait complete. 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.294+0000 7fd3d4e29740 1 -- start start 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 Processor -- start 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x5646745597d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674560d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564674557e90 con 0x56467454fe30 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56467446ecf0 con 0x564674554fa0 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x564674558010 con 0x56467454f440 2024-04-24T23:44:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3d4e29740 10 client.0 _ll_register_callbacks cb 0x56467446e6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47180/0 (socket says 172.21.15.107:47180) 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 -- 172.21.15.107:0/1303806910 learned_addr learned my addr 172.21.15.107:0/1303806910 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x5646745597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 -- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 msgr2=0x5646745597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x5646745597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674560d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 -- 172.21.15.107:0/1303806910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 msgr2=0x564674560d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674560d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3c3fff640 1 -- 172.21.15.107:0/1303806910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646742916b0 con 0x56467454fe30 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.295+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674560d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400f670 tx=0x7fd3b400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef71a3c25db4ee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b401d060 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3b400da20 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3bc09b360 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3b4024050 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 10 client.15240 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564674563e30 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc09b360 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b4016340 con 0x56467454fe30 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 10 client.15240 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 20 client.15240 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x564674566520 0x564674567990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 4 client.15240 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.296+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646744c0440 con 0x564674566520 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.297+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x564674566520 0x564674567990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.297+0000 7fd3aaffd640 10 client.15240 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.297+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x564674566520 0x564674567990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5deb8932ad7eef2 server_cookie=a1c34830306620fe in_seq=0 out_seq=0 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3aaffd640 1 -- 172.21.15.107:0/1303806910 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646744c0440 con 0x564674566520 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3aaffd640 10 client.15240 handle_command_reply: tid=0 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 client.15240 shutdown 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 msgr2=0x564674559d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400f670 tx=0x7fd3b400af90 comp rx=0 tx=0).stop 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 shutdown_connections 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x564674566520 0x564674567990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564674554fa0 0x564674560d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x564674559d10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3aaffd640 0 client.15240 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/1303806910 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56467454f440 0x5646745597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 >> 172.21.15.107:0/1303806910 conn(0x56467454aab0 msgr2=0x564674549ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 shutdown_connections 2024-04-24T23:44:43.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 1 -- 172.21.15.107:0/1303806910 wait complete. 2024-04-24T23:44:43.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.298+0000 7fd3d4e29740 20 client.15240 trim_cache size 0 max 16384 2024-04-24T23:44:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 Processor -- start 2024-04-24T23:44:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 -- start start 2024-04-24T23:44:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 0x564674560d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x564674549060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564674565cd0 con 0x56467454f440 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5646745663a0 con 0x56467454fe30 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x564674566a70 con 0x564674554fa0 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47192/0 (socket says 172.21.15.107:47192) 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3291306636 learned_addr learned my addr 172.21.15.107:0/3291306636 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x564674549060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3291306636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 msgr2=0x564674549060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 0x564674560d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x564674549060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3291306636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 msgr2=0x564674560d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 0x564674560d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 -- 172.21.15.107:0/3291306636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564674550210 con 0x56467454f440 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 0x564674560d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x564674549060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.316+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09b2f0 tx=0x7fd3bc0a40d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e14f7c3ecda51c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3c17fa640 1 -- 172.21.15.107:0/3291306636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc09ba60 con 0x56467454f440 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3c17fa640 1 -- 172.21.15.107:0/3291306636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc09f710 con 0x56467454f440 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3c17fa640 1 -- 172.21.15.107:0/3291306636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0a3b40 con 0x56467454f440 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- 172.21.15.107:0/3291306636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 msgr2=0x564674554860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09b2f0 tx=0x7fd3bc0a40d0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- 172.21.15.107:0/3291306636 shutdown_connections 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x564674549060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454fe30 0x564674560d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/3291306636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454f440 0x564674554860 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- 172.21.15.107:0/3291306636 >> 172.21.15.107:0/3291306636 conn(0x56467454aab0 msgr2=0x56467454ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- 172.21.15.107:0/3291306636 shutdown_connections 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- 172.21.15.107:0/3291306636 wait complete. 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.317+0000 7fd3d4e29740 1 -- start start 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 Processor -- start 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 0x56467455c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x56467455f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564674567810 con 0x56467454fe30 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x564674567140 con 0x56467454f440 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x564674567ee0 con 0x564674554fa0 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3d4e29740 10 client.0 _ll_register_callbacks cb 0x7fd3b400ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47204/0 (socket says 172.21.15.107:47204) 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 -- 172.21.15.107:0/2250016965 learned_addr learned my addr 172.21.15.107:0/2250016965 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 0x56467455c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 -- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 msgr2=0x56467455f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x56467455f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x56467455f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 -- 172.21.15.107:0/2250016965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 msgr2=0x56467455c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 0x56467455c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 -- 172.21.15.107:0/2250016965 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467455fde0 con 0x56467454fe30 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 0x56467455c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3cbfff640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x56467455f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3c3fff640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400f640 tx=0x7fd3b4005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829f4bf9c5eef426 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.318+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b402c070 con 0x56467454fe30 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3b4005040 con 0x56467454fe30 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3bc09f710 con 0x56467454fe30 2024-04-24T23:44:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3b401d020 con 0x56467454fe30 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 10 client.15246 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5646745cc4d0 con 0x56467454fe30 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3bc09f710 con 0x56467454fe30 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b400dc00 con 0x56467454fe30 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 10 client.15246 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 10 client.15246 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 20 client.15246 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5646745d0f80 0x5646745cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 4 client.15246 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "status"}) -- 0x5646744ba790 con 0x5646745d0f80 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.319+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5646745d0f80 0x5646745cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3aaffd640 10 client.15246 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3cb7fe640 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5646745d0f80 0x5646745cfc90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bbdfc9ab52e5dbfa server_cookie=75ea696c09b67e6a in_seq=0 out_seq=0 2024-04-24T23:44:43.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3aaffd640 1 -- 172.21.15.107:0/2250016965 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5646744ba790 con 0x5646745d0f80 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3aaffd640 10 client.15246 handle_command_reply: tid=0 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 client.15246 shutdown 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 msgr2=0x56467455c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400f640 tx=0x7fd3b4005b10 comp rx=0 tx=0).stop 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 shutdown_connections 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5646745d0f80 0x5646745cfc90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x564674554fa0 0x56467455f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3aaffd640 0 client.15246 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56467454fe30 0x56467455c7d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 --2- 172.21.15.107:0/2250016965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56467454f440 0x56467455c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 >> 172.21.15.107:0/2250016965 conn(0x56467454aab0 msgr2=0x56467454d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 shutdown_connections 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 1 -- 172.21.15.107:0/2250016965 wait complete. 2024-04-24T23:44:43.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.320+0000 7fd3d4e29740 20 client.15246 trim_cache size 0 max 16384 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 4, 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24451, 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 85.231107397000002, 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:44:43.329 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-04-24T23:44:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:44:43.330 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 136.27091224200001 2024-04-24T23:44:43.330 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:44:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 msgr2=0x7fd3b80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 0x7fd3b80a50e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x564674550690 tx=0x7fd3c402f940 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 msgr2=0x7fd3cc199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00d950 tx=0x7fd3bc00de10 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 shutdown_connections 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd3b80a2c20 0x7fd3b80a50e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3cc109640 0x7fd3cc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd3cc075700 0x7fd3cc19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 --2- 172.21.15.107:0/3436599058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3cc074120 0x7fd3cc199be0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 >> 172.21.15.107:0/3436599058 conn(0x7fd3cc0fcd20 msgr2=0x7fd3cc1055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 shutdown_connections 2024-04-24T23:44:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.321+0000 7fd3d2038640 1 -- 172.21.15.107:0/3436599058 wait complete. 2024-04-24T23:44:43.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.429+0000 7fbc6ca64640 1 -- 172.21.15.107:0/4289416541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 msgr2=0x7fbc68100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.429+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/4289416541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc68100ac0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009900 tx=0x7fbc6002a410 comp rx=0 tx=0).stop 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- 172.21.15.107:0/4289416541 shutdown_connections 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/4289416541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 0x7fbc68108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/4289416541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 0x7fbc68101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/4289416541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc68100ac0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- 172.21.15.107:0/4289416541 >> 172.21.15.107:0/4289416541 conn(0x7fbc680fbc30 msgr2=0x7fbc680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- 172.21.15.107:0/4289416541 shutdown_connections 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- 172.21.15.107:0/4289416541 wait complete. 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 Processor -- start 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- start start 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 0x7fbc6819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 0x7fbc6819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.430+0000 7fbc6ca64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6810c2f0 con 0x7fbc681006a0 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6ca64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6810c170 con 0x7fbc68102250 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6ca64640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6810c470 con 0x7fbc68101890 2024-04-24T23:44:43.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47232/0 (socket says 172.21.15.107:47232) 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 -- 172.21.15.107:0/2580787225 learned_addr learned my addr 172.21.15.107:0/2580787225 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 0x7fbc6819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 0x7fbc6819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 -- 172.21.15.107:0/2580787225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 msgr2=0x7fbc6819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 0x7fbc6819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 -- 172.21.15.107:0/2580787225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 msgr2=0x7fbc6819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 0x7fbc6819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 -- 172.21.15.107:0/2580787225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6819f6e0 con 0x7fbc681006a0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbc60031610 tx=0x7fbc60004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc577fe640 1 -- 172.21.15.107:0/2580787225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6003f030 con 0x7fbc681006a0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6819f910 con 0x7fbc681006a0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc577fe640 1 -- 172.21.15.107:0/2580787225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc60003ad0 con 0x7fbc681006a0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc577fe640 1 -- 172.21.15.107:0/2580787225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60035070 con 0x7fbc681006a0 2024-04-24T23:44:43.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.431+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6819fe00 con 0x7fbc681006a0 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.433+0000 7fbc577fe640 1 -- 172.21.15.107:0/2580787225 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbc60042030 con 0x7fbc681006a0 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 Processor -- start 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 -- start start 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba7819930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba781c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ba7777f00 con 0x560ba7814440 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ba7720750 con 0x560ba7819fa0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6f773740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ba7773ec0 con 0x560ba7814e30 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc577fe640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 0x7fbc500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47242/0 (socket says 172.21.15.107:47242) 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3759126443 learned_addr learned my addr 172.21.15.107:0/3759126443 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 0x7fbc500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba781c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc577fe640 1 -- 172.21.15.107:0/2580787225 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc600ef920 con 0x7fbc681006a0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba7819930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3759126443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 msgr2=0x560ba7819930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba7819930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.434+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3759126443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 msgr2=0x560ba781c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba781c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3759126443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba748c2d0 con 0x560ba7814440 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbc580138b0 tx=0x7fbc58013d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae2fe478dbfda3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc54ff9640 1 -- 172.21.15.107:0/3759126443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58023020 con 0x560ba7814440 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 0x7fbc500a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbc58024000 tx=0x7fbc58022000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc54ff9640 1 -- 172.21.15.107:0/3759126443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc58010430 con 0x560ba7814440 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.435+0000 7fbc54ff9640 1 -- 172.21.15.107:0/3759126443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5801e620 con 0x560ba7814440 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- 172.21.15.107:0/3759126443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 msgr2=0x560ba7814860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbc580138b0 tx=0x7fbc58013d70 comp rx=0 tx=0).stop 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- 172.21.15.107:0/3759126443 shutdown_connections 2024-04-24T23:44:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba781c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba7819930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3759126443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814440 0x560ba7814860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- 172.21.15.107:0/3759126443 >> 172.21.15.107:0/3759126443 conn(0x560ba780fab0 msgr2=0x560ba780feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- 172.21.15.107:0/3759126443 shutdown_connections 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- 172.21.15.107:0/3759126443 wait complete. 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- start start 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 Processor -- start 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba781e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7819fa0 0x560ba7825db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ba781cf50 con 0x560ba7814e30 2024-04-24T23:44:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ba7733cf0 con 0x560ba7814440 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ba781d0d0 con 0x560ba7819fa0 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc6f773740 10 client.0 _ll_register_callbacks cb 0x560ba77336e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47246/0 (socket says 172.21.15.107:47246) 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.436+0000 7fbc65d9b640 1 -- 172.21.15.107:0/3963496652 learned_addr learned my addr 172.21.15.107:0/3963496652 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba781e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 -- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7819fa0 msgr2=0x560ba7825db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7819fa0 0x560ba7825db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 -- 172.21.15.107:0/3963496652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 msgr2=0x560ba781e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba781e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 -- 172.21.15.107:0/3963496652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba75566b0 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00fa30 tx=0x7fbc5c00fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86cf14c2efb020b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00dd90 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc5c01b480 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc58010430 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbc5c01b680 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 10 client.15258 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ba7828e60 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc58010430 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbc5c010930 con 0x560ba7814e30 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 10 client.15258 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 10 client.15258 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 20 client.15258 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.437+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba782b5c0 0x560ba782ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.438+0000 7fbc6f773740 4 client.15258 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.438+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ba7785440 con 0x560ba782b5c0 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.438+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba782b5c0 0x560ba782ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.438+0000 7fbc4dffb640 10 client.15258 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.438+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba782b5c0 0x560ba782ca30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=98e324686d56ab server_cookie=475e364d5706690f in_seq=0 out_seq=0 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc4dffb640 1 -- 172.21.15.107:0/3963496652 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ba7785440 con 0x560ba782b5c0 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc4dffb640 10 client.15258 handle_command_reply: tid=0 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 client.15258 shutdown 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 msgr2=0x560ba781ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00fa30 tx=0x7fbc5c00fef0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 shutdown_connections 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba782b5c0 0x560ba782ca30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7819fa0 0x560ba7825db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba781ed70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc4dffb640 0 client.15258 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3963496652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba781e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 >> 172.21.15.107:0/3963496652 conn(0x560ba780fab0 msgr2=0x560ba780eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 shutdown_connections 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 1 -- 172.21.15.107:0/3963496652 wait complete. 2024-04-24T23:44:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.439+0000 7fbc6f773740 20 client.15258 trim_cache size 0 max 16384 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 Processor -- start 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 -- start start 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba780a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba781e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ba782ad70 con 0x560ba7819fa0 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ba782b440 con 0x560ba7814440 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.457+0000 7fbc6f773740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ba782bb10 con 0x560ba7814e30 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47262/0 (socket says 172.21.15.107:47262) 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3717238792 learned_addr learned my addr 172.21.15.107:0/3717238792 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba781e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3717238792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 msgr2=0x560ba781e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba781e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba780a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3717238792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 msgr2=0x560ba780a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba780a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 -- 172.21.15.107:0/3717238792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba7815cb0 con 0x560ba7819fa0 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc6659c640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba780a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbc58013760 tx=0x7fbc58013790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32079dfc5c45437 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc4ffff640 1 -- 172.21.15.107:0/3717238792 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58023030 con 0x560ba7819fa0 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc4ffff640 1 -- 172.21.15.107:0/3717238792 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc580122a0 con 0x560ba7819fa0 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.458+0000 7fbc4ffff640 1 -- 172.21.15.107:0/3717238792 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc580035f0 con 0x560ba7819fa0 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- 172.21.15.107:0/3717238792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 msgr2=0x560ba781e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbc58013760 tx=0x7fbc58013790 comp rx=0 tx=0).stop 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- 172.21.15.107:0/3717238792 shutdown_connections 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7819fa0 0x560ba781e980 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814e30 0x560ba781e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- 172.21.15.107:0/3717238792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7814440 0x560ba780a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- 172.21.15.107:0/3717238792 >> 172.21.15.107:0/3717238792 conn(0x560ba780fab0 msgr2=0x560ba780fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- 172.21.15.107:0/3717238792 shutdown_connections 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- 172.21.15.107:0/3717238792 wait complete. 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- start start 2024-04-24T23:44:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 Processor -- start 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814440 0x560ba788f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba7894350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ba782c8b0 con 0x560ba7814e30 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ba782c1e0 con 0x560ba7819fa0 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ba782cf80 con 0x560ba7814440 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6f773740 10 client.0 _ll_register_callbacks cb 0x7fbc5c00f5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47266/0 (socket says 172.21.15.107:47266) 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc65d9b640 1 -- 172.21.15.107:0/2276748259 learned_addr learned my addr 172.21.15.107:0/2276748259 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.459+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814440 0x560ba788f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba7894350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 -- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814440 msgr2=0x560ba788f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814440 0x560ba788f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 -- 172.21.15.107:0/2276748259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 msgr2=0x560ba7894350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba7894350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 -- 172.21.15.107:0/2276748259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba7894b80 con 0x560ba7814e30 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc66d9d640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba7894350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc65d9b640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c010e20 tx=0x7fbc5c00d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da0229621fdf6d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004bc0 con 0x560ba7814e30 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc5c004d60 con 0x560ba7814e30 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc580122a0 con 0x560ba7814e30 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbc5c01b820 con 0x560ba7814e30 2024-04-24T23:44:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 10 client.15264 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ba7895ec0 con 0x560ba7814e30 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc580122a0 con 0x560ba7814e30 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbc5c005040 con 0x560ba7814e30 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 10 client.15264 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 10 client.15264 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.460+0000 7fbc6f773740 20 client.15264 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba789ab50 0x560ba7896f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 4 client.15264 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "status"}) -- 0x560ba777f790 con 0x560ba789ab50 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba789ab50 0x560ba7896f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc4dffb640 10 client.15264 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:44:43.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6659c640 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba789ab50 0x560ba7896f90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39db03624c0b209e server_cookie=f3bf2ebc4a31c426 in_seq=0 out_seq=0 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc4dffb640 1 -- 172.21.15.107:0/2276748259 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x560ba777f790 con 0x560ba789ab50 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc4dffb640 10 client.15264 handle_command_reply: tid=0 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 client.15264 shutdown 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 msgr2=0x560ba7891ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c010e20 tx=0x7fbc5c00d850 comp rx=0 tx=0).stop 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 shutdown_connections 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x560ba789ab50 0x560ba7896f90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ba7819fa0 0x560ba7894350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ba7814e30 0x560ba7891ca0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc4dffb640 0 client.15264 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.461+0000 7fbc6f773740 1 --2- 172.21.15.107:0/2276748259 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ba7814440 0x560ba788f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 >> 172.21.15.107:0/2276748259 conn(0x560ba780fab0 msgr2=0x560ba780a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 shutdown_connections 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6f773740 1 -- 172.21.15.107:0/2276748259 wait complete. 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6f773740 20 client.15264 trim_cache size 0 max 16384 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 3, 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24463, 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:44:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 86.382602302999999, 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 133.49034885 2024-04-24T23:44:43.473 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:44:43.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 msgr2=0x7fbc500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 0x7fbc500a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbc58024000 tx=0x7fbc58022000 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 msgr2=0x7fbc6806a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbc60031610 tx=0x7fbc60004270 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 shutdown_connections 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc500a2c20 0x7fbc500a50e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc68102250 0x7fbc6819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc68101890 0x7fbc6819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 --2- 172.21.15.107:0/2580787225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc681006a0 0x7fbc6806a8f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 >> 172.21.15.107:0/2580787225 conn(0x7fbc680fbc30 msgr2=0x7fbc68104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 shutdown_connections 2024-04-24T23:44:43.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.462+0000 7fbc6ca64640 1 -- 172.21.15.107:0/2580787225 wait complete. 2024-04-24T23:44:43.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-24T23:44:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:43 smithi195 ceph-mon[33019]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:43 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2632565029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:43 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3460872301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:43 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3376975388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 -- 172.21.15.107:0/2886248599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 msgr2=0x7fccdc109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 --2- 172.21.15.107:0/2886248599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc109bd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009ca0 tx=0x7fccd402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 -- 172.21.15.107:0/2886248599 shutdown_connections 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 --2- 172.21.15.107:0/2886248599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc109bd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 --2- 172.21.15.107:0/2886248599 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc102980 0x7fccdc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 --2- 172.21.15.107:0/2886248599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc101790 0x7fccdc101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 -- 172.21.15.107:0/2886248599 >> 172.21.15.107:0/2886248599 conn(0x7fccdc0fcd20 msgr2=0x7fccdc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 -- 172.21.15.107:0/2886248599 shutdown_connections 2024-04-24T23:44:43.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.574+0000 7fcce0d84640 1 -- 172.21.15.107:0/2886248599 wait complete. 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 Processor -- start 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 -- start start 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 0x7fccdc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc102980 0x7fccdc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10d3f0 con 0x7fccdc103340 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10d270 con 0x7fccdc102980 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fcce0d84640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10d570 con 0x7fccdc101790 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47290/0 (socket says 172.21.15.107:47290) 2024-04-24T23:44:43.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 -- 172.21.15.107:0/629472567 learned_addr learned my addr 172.21.15.107:0/629472567 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccda59c640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 0x7fccdc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccd3fff640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc102980 0x7fccdc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 -- 172.21.15.107:0/629472567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 msgr2=0x7fccdc0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 0x7fccdc0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 -- 172.21.15.107:0/629472567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc102980 msgr2=0x7fccdc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc102980 0x7fccdc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 -- 172.21.15.107:0/629472567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc074900 con 0x7fccdc103340 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccda59c640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 0x7fccdc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.575+0000 7fccdad9d640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fccd4005e30 tx=0x7fccd4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.576+0000 7fccd37fe640 1 -- 172.21.15.107:0/629472567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4003c30 con 0x7fccdc103340 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.576+0000 7fccd37fe640 1 -- 172.21.15.107:0/629472567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccd4003dd0 con 0x7fccdc103340 2024-04-24T23:44:43.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.576+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccdc1a1b50 con 0x7fccdc103340 2024-04-24T23:44:43.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.576+0000 7fccd37fe640 1 -- 172.21.15.107:0/629472567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd402fde0 con 0x7fccdc103340 2024-04-24T23:44:43.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.576+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccdc1a20a0 con 0x7fccdc103340 2024-04-24T23:44:43.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:43 smithi170 ceph-mon[32714]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:44:43.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:43 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2632565029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:44:43.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:43 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3460872301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:43 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3376975388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:44:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fccd37fe640 1 -- 172.21.15.107:0/629472567 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fccd4004050 con 0x7fccdc103340 2024-04-24T23:44:43.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 Processor -- start 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 -- start start 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124440 0x55df1c124860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c12c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df1c087f00 con 0x55df1c124e30 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55df1c030750 con 0x55df1c124440 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fcce3b68740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55df1c083ec0 con 0x55df1c129fa0 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.578+0000 7fccd37fe640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 0x7fccc80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47300/0 (socket says 172.21.15.107:47300) 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 -- 172.21.15.107:0/3947844314 learned_addr learned my addr 172.21.15.107:0/3947844314 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccdad9d640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 0x7fccc80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd3fff640 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124440 0x55df1c124860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd37fe640 1 -- 172.21.15.107:0/629472567 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fccd40f09e0 con 0x7fccdc103340 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd3fff640 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c12c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 -- 172.21.15.107:0/3947844314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 msgr2=0x55df1c12c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c12c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 -- 172.21.15.107:0/3947844314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124440 msgr2=0x55df1c124860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124440 0x55df1c124860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 -- 172.21.15.107:0/3947844314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df1bd9c2d0 con 0x55df1c124e30 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccda59c640 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00cc50 tx=0x7fcccc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454b49b20e7ab121 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccdad9d640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 0x7fccc80a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccd40b9410 tx=0x7fccd403d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd0ff9640 1 -- 172.21.15.107:0/3947844314 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc014070 con 0x55df1c124e30 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd0ff9640 1 -- 172.21.15.107:0/3947844314 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcccc0047a0 con 0x55df1c124e30 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.579+0000 7fccd0ff9640 1 -- 172.21.15.107:0/3947844314 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0107d0 con 0x55df1c124e30 2024-04-24T23:44:43.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- 172.21.15.107:0/3947844314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 msgr2=0x55df1c129930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00cc50 tx=0x7fcccc00e720 comp rx=0 tx=0).stop 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- 172.21.15.107:0/3947844314 shutdown_connections 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c12c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c129930 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 --2- 172.21.15.107:0/3947844314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124440 0x55df1c124860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- 172.21.15.107:0/3947844314 >> 172.21.15.107:0/3947844314 conn(0x55df1c11fab0 msgr2=0x55df1c11feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- 172.21.15.107:0/3947844314 shutdown_connections 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- 172.21.15.107:0/3947844314 wait complete. 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 -- start start 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.580+0000 7fcce3b68740 1 Processor -- start 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c12eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df1c12cf50 con 0x55df1c129fa0 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55df1c043cf0 con 0x55df1c124e30 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55df1c12d0d0 con 0x55df1c124440 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 10 client.0 _ll_register_callbacks cb 0x55df1c0436e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47304/0 (socket says 172.21.15.107:47304) 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 -- 172.21.15.107:0/1264833429 learned_addr learned my addr 172.21.15.107:0/1264833429 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccda59c640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c12eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 -- 172.21.15.107:0/1264833429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 msgr2=0x55df1c12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 -- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 msgr2=0x55df1c12eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c12eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 -- 172.21.15.107:0/1264833429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df1be666b0 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccd3fff640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fccc4014e40 tx=0x7fccc400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66dc75b08bf798d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc40141d0 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fccc4004030 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.581+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcccc0047a0 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fccc4028050 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 10 client.15276 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df1c138e30 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcccc0047a0 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fccc4003bf0 con 0x55df1c129fa0 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 10 client.15276 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 10 client.15276 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 20 client.15276 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c13b4c0 0x55df1c13c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 4 client.15276 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df1c095440 con 0x55df1c13b4c0 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.582+0000 7fccda59c640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c13b4c0 0x55df1c13c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.583+0000 7fccc1ffb640 10 client.15276 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.583+0000 7fccda59c640 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c13b4c0 0x55df1c13c930 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78f90ab3e878f85d server_cookie=6836d5c22f3f8ed in_seq=0 out_seq=0 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.583+0000 7fccc1ffb640 1 -- 172.21.15.107:0/1264833429 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df1c095440 con 0x55df1c13b4c0 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.583+0000 7fccc1ffb640 10 client.15276 handle_command_reply: tid=0 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.583+0000 7fcce3b68740 1 client.15276 shutdown 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 msgr2=0x55df1c135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fccc4014e40 tx=0x7fccc400cb70 comp rx=0 tx=0).stop 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 shutdown_connections 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c13b4c0 0x55df1c13c930 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c129fa0 0x55df1c135de0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c12eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 --2- 172.21.15.107:0/1264833429 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fccc1ffb640 0 client.15276 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 >> 172.21.15.107:0/1264833429 conn(0x55df1c11fab0 msgr2=0x55df1c11eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 shutdown_connections 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 1 -- 172.21.15.107:0/1264833429 wait complete. 2024-04-24T23:44:43.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.584+0000 7fcce3b68740 20 client.15276 trim_cache size 0 max 16384 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.601+0000 7fcce3b68740 1 Processor -- start 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.601+0000 7fcce3b68740 1 -- start start 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c129fa0 0x55df1c12d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13ac70 con 0x55df1c124e30 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13b340 con 0x55df1c129fa0 2024-04-24T23:44:43.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fcce3b68740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13ba10 con 0x55df1c124440 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47312/0 (socket says 172.21.15.107:47312) 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 -- 172.21.15.107:0/2333327883 learned_addr learned my addr 172.21.15.107:0/2333327883 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccd3fff640 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c129fa0 0x55df1c12d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 -- 172.21.15.107:0/2333327883 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 msgr2=0x55df1c12eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 -- 172.21.15.107:0/2333327883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c129fa0 msgr2=0x55df1c12d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c129fa0 0x55df1c12d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 -- 172.21.15.107:0/2333327883 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df1c1239d0 con 0x55df1c124e30 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccda59c640 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00cba0 tx=0x7fcccc00c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8f9dfa4b5e5b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccc3fff640 1 -- 172.21.15.107:0/2333327883 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc02e070 con 0x55df1c124e30 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccc3fff640 1 -- 172.21.15.107:0/2333327883 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcccc004bd0 con 0x55df1c124e30 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.602+0000 7fccc3fff640 1 -- 172.21.15.107:0/2333327883 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc014040 con 0x55df1c124e30 2024-04-24T23:44:43.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- 172.21.15.107:0/2333327883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 msgr2=0x55df1c135d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00cba0 tx=0x7fcccc00c570 comp rx=0 tx=0).stop 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- 172.21.15.107:0/2333327883 shutdown_connections 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c129fa0 0x55df1c12d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124e30 0x55df1c135d00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- 172.21.15.107:0/2333327883 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c124440 0x55df1c12eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- 172.21.15.107:0/2333327883 >> 172.21.15.107:0/2333327883 conn(0x55df1c11fab0 msgr2=0x55df1c12e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- 172.21.15.107:0/2333327883 shutdown_connections 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- 172.21.15.107:0/2333327883 wait complete. 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- start start 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 Processor -- start 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c134480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c1349c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13c7b0 con 0x55df1c124440 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13c0e0 con 0x55df1c124e30 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55df1c13ce80 con 0x55df1c129fa0 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fcce3b68740 10 client.0 _ll_register_callbacks cb 0x7fccc4014cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47324/0 (socket says 172.21.15.107:47324) 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.603+0000 7fccd3fff640 1 -- 172.21.15.107:0/568689261 learned_addr learned my addr 172.21.15.107:0/568689261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 -- 172.21.15.107:0/568689261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 msgr2=0x55df1c1349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c1349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 -- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 msgr2=0x55df1c134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 -- 172.21.15.107:0/568689261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df1c1351f0 con 0x55df1c124440 2024-04-24T23:44:43.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccd3fff640 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fccc4005fd0 tx=0x7fccc400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9324c71cda87c7dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4031060 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fccc400f820 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcccc004bd0 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fccc4038050 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df1c1a16d0 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcccc004bd0 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fccc4005030 con 0x55df1c124440 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 10 client.15282 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 20 client.15282 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c1a62d0 0x55df1c1a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 4 client.15282 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:44:43.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.604+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "status"}) -- 0x55df1c08f790 con 0x55df1c1a62d0 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccda59c640 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c1a62d0 0x55df1c1a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccc1ffb640 10 client.15282 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccda59c640 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c1a62d0 0x55df1c1a4fe0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c56426e34eed2c server_cookie=b4ed441a367205f0 in_seq=0 out_seq=0 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccc1ffb640 1 -- 172.21.15.107:0/568689261 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55df1c08f790 con 0x55df1c1a62d0 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccc1ffb640 10 client.15282 handle_command_reply: tid=0 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 client.15282 shutdown 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 msgr2=0x55df1c133f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fccc4005fd0 tx=0x7fccc400cf90 comp rx=0 tx=0).stop 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 shutdown_connections 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55df1c1a62d0 0x55df1c1a4fe0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55df1c129fa0 0x55df1c1349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55df1c124e30 0x55df1c134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fccc1ffb640 0 client.15282 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 --2- 172.21.15.107:0/568689261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df1c124440 0x55df1c133f40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 >> 172.21.15.107:0/568689261 conn(0x55df1c11fab0 msgr2=0x55df1c12e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 shutdown_connections 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 1 -- 172.21.15.107:0/568689261 wait complete. 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.605+0000 7fcce3b68740 20 client.15282 trim_cache size 0 max 16384 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24464, 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:44:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 87.533908928000002, 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 130.763964491 2024-04-24T23:44:43.615 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 msgr2=0x7fccc80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 0x7fccc80a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccd40b9410 tx=0x7fccd403d040 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 msgr2=0x7fccdc074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fccd4005e30 tx=0x7fccd4031ee0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 shutdown_connections 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fccc80a2c20 0x7fccc80a50e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccdc103340 0x7fccdc074070 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccdc102980 0x7fccdc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 --2- 172.21.15.107:0/629472567 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fccdc101790 0x7fccdc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 >> 172.21.15.107:0/629472567 conn(0x7fccdc0fcd20 msgr2=0x7fccdc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 shutdown_connections 2024-04-24T23:44:43.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.606+0000 7fcce0d84640 1 -- 172.21.15.107:0/629472567 wait complete. 2024-04-24T23:44:43.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-24T23:44:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 -- 172.21.15.107:0/2514022834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 msgr2=0x7f58a8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 --2- 172.21.15.107:0/2514022834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a8108ae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009ca0 tx=0x7f58a002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 -- 172.21.15.107:0/2514022834 shutdown_connections 2024-04-24T23:44:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 --2- 172.21.15.107:0/2514022834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a8108ae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 --2- 172.21.15.107:0/2514022834 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a8101890 0x7f58a8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 --2- 172.21.15.107:0/2514022834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a81006a0 0x7f58a8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 -- 172.21.15.107:0/2514022834 >> 172.21.15.107:0/2514022834 conn(0x7f58a80fbc10 msgr2=0x7f58a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 -- 172.21.15.107:0/2514022834 shutdown_connections 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.715+0000 7f58acea5640 1 -- 172.21.15.107:0/2514022834 wait complete. 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 Processor -- start 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 -- start start 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a81006a0 0x7f58a81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a8101890 0x7f58a8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f58a810c320 con 0x7f58a8102250 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f58a810c1a0 con 0x7f58a8101890 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58acea5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f58a810c4a0 con 0x7f58a81006a0 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47340/0 (socket says 172.21.15.107:47340) 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 -- 172.21.15.107:0/335352562 learned_addr learned my addr 172.21.15.107:0/335352562 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f589ffff640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a8101890 0x7f58a8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a659c640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a81006a0 0x7f58a81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 -- 172.21.15.107:0/335352562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a81006a0 msgr2=0x7f58a81979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a81006a0 0x7f58a81979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 -- 172.21.15.107:0/335352562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a8101890 msgr2=0x7f58a8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a8101890 0x7f58a8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 -- 172.21.15.107:0/335352562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a819f5f0 con 0x7f58a8102250 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.716+0000 7f58a6d9d640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58a0005e30 tx=0x7f58a0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.717+0000 7f589f7fe640 1 -- 172.21.15.107:0/335352562 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0003c30 con 0x7f58a8102250 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.717+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a819f880 con 0x7f58a8102250 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.717+0000 7f589f7fe640 1 -- 172.21.15.107:0/335352562 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58a0003dd0 con 0x7f58a8102250 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.717+0000 7f589f7fe640 1 -- 172.21.15.107:0/335352562 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a002fde0 con 0x7f58a8102250 2024-04-24T23:44:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.717+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a819fe20 con 0x7f58a8102250 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.719+0000 7f589f7fe640 1 -- 172.21.15.107:0/335352562 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f58a0004050 con 0x7f58a8102250 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.719+0000 7f58afbb4740 1 Processor -- start 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.719+0000 7f58afbb4740 1 -- start start 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.719+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f8d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f92930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58afbb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a48ef0f00 con 0x561a48f92fa0 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58afbb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a48e99750 con 0x561a48f8de30 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58afbb4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a48eecec0 con 0x561a48f8d440 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589f7fe640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 0x7f58940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47344/0 (socket says 172.21.15.107:47344) 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 -- 172.21.15.107:0/819864560 learned_addr learned my addr 172.21.15.107:0/819864560 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58a6d9d640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 0x7f58940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f8d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58a659c640 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f92930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589f7fe640 1 -- 172.21.15.107:0/335352562 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58a00f0750 con 0x7f58a8102250 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 -- 172.21.15.107:0/819864560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 msgr2=0x561a48f8d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f8d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 -- 172.21.15.107:0/819864560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 msgr2=0x561a48f92930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f92930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 -- 172.21.15.107:0/819864560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a48c052d0 con 0x561a48f92fa0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589ffff640 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x561a48f8e690 tx=0x7f58900041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef3b9f42a9e4cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f58a6d9d640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 0x7f58940a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f58a00b95d0 tx=0x7f58a00e7790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589cff9640 1 -- 172.21.15.107:0/819864560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890005020 con 0x561a48f92fa0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589cff9640 1 -- 172.21.15.107:0/819864560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58900047a0 con 0x561a48f92fa0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.720+0000 7f589cff9640 1 -- 172.21.15.107:0/819864560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58900126c0 con 0x561a48f92fa0 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- 172.21.15.107:0/819864560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 msgr2=0x561a48f95430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x561a48f8e690 tx=0x7f58900041f0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- 172.21.15.107:0/819864560 shutdown_connections 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f95430 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f92930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 --2- 172.21.15.107:0/819864560 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f8d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- 172.21.15.107:0/819864560 >> 172.21.15.107:0/819864560 conn(0x561a48f88ab0 msgr2=0x561a48f88eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- 172.21.15.107:0/819864560 shutdown_connections 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- 172.21.15.107:0/819864560 wait complete. 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.721+0000 7f58afbb4740 1 -- start start 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 Processor -- start 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8de30 0x561a48f97df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f92fa0 0x561a48f9ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a48f95f50 con 0x561a48f8d440 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a48eaccf0 con 0x561a48f92fa0 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a48f960d0 con 0x561a48f8de30 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58afbb4740 10 client.0 _ll_register_callbacks cb 0x561a48eac6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47346/0 (socket says 172.21.15.107:47346) 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 -- 172.21.15.107:0/3181405151 learned_addr learned my addr 172.21.15.107:0/3181405151 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f58a659c640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8de30 0x561a48f97df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f92fa0 0x561a48f9ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 -- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8de30 msgr2=0x561a48f97df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8de30 0x561a48f97df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 -- 172.21.15.107:0/3181405151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f92fa0 msgr2=0x561a48f9ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f92fa0 0x561a48f9ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 -- 172.21.15.107:0/3181405151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a48ccf6b0 con 0x561a48f8d440 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f589ffff640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f58900057c0 tx=0x7f5890005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1c6975be2e3ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.722+0000 7f588dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890005200 con 0x561a48f8d440 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58900047a0 con 0x561a48f8d440 2024-04-24T23:44:43.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58900053a0 con 0x561a48f8d440 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5890012690 con 0x561a48f8d440 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 10 client.15294 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a48fa1e40 con 0x561a48f8d440 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58900047a0 con 0x561a48f8d440 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f589001e860 con 0x561a48f8d440 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 10 client.15294 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 10 client.15294 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 20 client.15294 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a48fa4260 0x561a48fa56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 4 client.15294 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a48efe440 con 0x561a48fa4260 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.723+0000 7f58a659c640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a48fa4260 0x561a48fa56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.724+0000 7f588dffb640 10 client.15294 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.724+0000 7f58a659c640 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a48fa4260 0x561a48fa56d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2850ed5690d88b28 server_cookie=ea14b01dc01dc345 in_seq=0 out_seq=0 2024-04-24T23:44:43.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f588dffb640 1 -- 172.21.15.107:0/3181405151 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a48efe440 con 0x561a48fa4260 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f588dffb640 10 client.15294 handle_command_reply: tid=0 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 client.15294 shutdown 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 msgr2=0x561a48f978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f58900057c0 tx=0x7f5890005710 comp rx=0 tx=0).stop 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 shutdown_connections 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a48fa4260 0x561a48fa56d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f92fa0 0x561a48f9ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f588dffb640 0 client.15294 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8de30 0x561a48f97df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3181405151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8d440 0x561a48f978b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 >> 172.21.15.107:0/3181405151 conn(0x561a48f88ab0 msgr2=0x561a48f87ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 shutdown_connections 2024-04-24T23:44:43.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 1 -- 172.21.15.107:0/3181405151 wait complete. 2024-04-24T23:44:43.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.725+0000 7f58afbb4740 20 client.15294 trim_cache size 0 max 16384 2024-04-24T23:44:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 Processor -- start 2024-04-24T23:44:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 -- start start 2024-04-24T23:44:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8d440 0x561a48f97bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 0x561a48f87250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa3a10 con 0x561a48f8de30 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa40e0 con 0x561a48f8d440 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58afbb4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa47b0 con 0x561a48f92fa0 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47360/0 (socket says 172.21.15.107:47360) 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 -- 172.21.15.107:0/3528493226 learned_addr learned my addr 172.21.15.107:0/3528493226 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 -- 172.21.15.107:0/3528493226 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 msgr2=0x561a48f87250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f589ffff640 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 0x561a48f87250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 0x561a48f87250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 -- 172.21.15.107:0/3528493226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8d440 msgr2=0x561a48f97bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.743+0000 7f58a659c640 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8d440 0x561a48f97bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58a659c640 1 -- 172.21.15.107:0/3528493226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a48f95e30 con 0x561a48f8de30 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f589ffff640 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 0x561a48f87250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58a659c640 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f589800ac90 tx=0x7f5898011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5436cbf862387c50 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f588ffff640 1 -- 172.21.15.107:0/3528493226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898004250 con 0x561a48f8de30 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f588ffff640 1 -- 172.21.15.107:0/3528493226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5898022070 con 0x561a48f8de30 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f588ffff640 1 -- 172.21.15.107:0/3528493226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898003d30 con 0x561a48f8de30 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 -- 172.21.15.107:0/3528493226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 msgr2=0x561a48f9edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f589800ac90 tx=0x7f5898011810 comp rx=0 tx=0).stop 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 -- 172.21.15.107:0/3528493226 shutdown_connections 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f92fa0 0x561a48f87250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f8de30 0x561a48f9edc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 --2- 172.21.15.107:0/3528493226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8d440 0x561a48f97bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 -- 172.21.15.107:0/3528493226 >> 172.21.15.107:0/3528493226 conn(0x561a48f88ab0 msgr2=0x561a48f974c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 -- 172.21.15.107:0/3528493226 shutdown_connections 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.744+0000 7f58afbb4740 1 -- 172.21.15.107:0/3528493226 wait complete. 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 -- start start 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 Processor -- start 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f9e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f9ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa5550 con 0x561a48f92fa0 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa4e80 con 0x561a48f8de30 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a48fa5c20 con 0x561a48f8d440 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58afbb4740 10 client.0 _ll_register_callbacks cb 0x7f5890003cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47366/0 (socket says 172.21.15.107:47366) 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 -- 172.21.15.107:0/1544678696 learned_addr learned my addr 172.21.15.107:0/1544678696 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f9e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f58a659c640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f9ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 -- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 msgr2=0x561a48f9e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f9e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 -- 172.21.15.107:0/1544678696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 msgr2=0x561a48f9ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f9ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.745+0000 7f589ffff640 1 -- 172.21.15.107:0/1544678696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a48f9bb40 con 0x561a48f92fa0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f589ffff640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5890004690 tx=0x7f5890025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d60216ecee72f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890004120 con 0x561a48f92fa0 2024-04-24T23:44:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5890012a30 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5898022070 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5890003a80 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a4900ae80 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5898022070 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5890012ca0 con 0x561a48f92fa0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 10 client.15300 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 20 client.15300 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a4900fc40 0x561a4900c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 4 client.15300 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.746+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "status"}) -- 0x561a48ef8790 con 0x561a4900fc40 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58a659c640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a4900fc40 0x561a4900c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f588dffb640 10 client.15300 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58a659c640 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a4900fc40 0x561a4900c080 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f332a65b37c8b48c server_cookie=4bc53b70546f1457 in_seq=0 out_seq=0 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f588dffb640 1 -- 172.21.15.107:0/1544678696 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561a48ef8790 con 0x561a4900fc40 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f588dffb640 10 client.15300 handle_command_reply: tid=0 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 client.15300 shutdown 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 msgr2=0x561a48f9b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5890004690 tx=0x7f5890025040 comp rx=0 tx=0).stop 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 shutdown_connections 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a48f92fa0 0x561a48f9b3b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a4900fc40 0x561a4900c080 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a48f8de30 0x561a48f9ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 --2- 172.21.15.107:0/1544678696 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a48f8d440 0x561a48f9e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f588dffb640 0 client.15300 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:44:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 >> 172.21.15.107:0/1544678696 conn(0x561a48f88ab0 msgr2=0x561a48f8b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 shutdown_connections 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.747+0000 7f58afbb4740 1 -- 172.21.15.107:0/1544678696 wait complete. 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58afbb4740 20 client.15300 trim_cache size 0 max 16384 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24493, 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:44:43.757 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 88.685897933999996, 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 128.037571364 2024-04-24T23:44:43.758 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 msgr2=0x7f58940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 0x7f58940a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f58a00b95d0 tx=0x7f58a00e7790 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 msgr2=0x7f58a819ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58a0005e30 tx=0x7f58a0031ee0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 shutdown_connections 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f58940a2c20 0x7f58940a50e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58a8102250 0x7f58a819ef80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58a8101890 0x7f58a8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 --2- 172.21.15.107:0/335352562 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f58a81006a0 0x7f58a81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 >> 172.21.15.107:0/335352562 conn(0x7f58a80fbc10 msgr2=0x7f58a8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 shutdown_connections 2024-04-24T23:44:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.748+0000 7f58acea5640 1 -- 172.21.15.107:0/335352562 wait complete. 2024-04-24T23:44:43.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 -- 172.21.15.107:0/4095891071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51381006a0 msgr2=0x7f5138100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 --2- 172.21.15.107:0/4095891071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51381006a0 0x7f5138100ac0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f512c009900 tx=0x7f512c02a410 comp rx=0 tx=0).stop 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 -- 172.21.15.107:0/4095891071 shutdown_connections 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 --2- 172.21.15.107:0/4095891071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 0x7f5138108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 --2- 172.21.15.107:0/4095891071 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5138101890 0x7f5138101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 --2- 172.21.15.107:0/4095891071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51381006a0 0x7f5138100ac0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 -- 172.21.15.107:0/4095891071 >> 172.21.15.107:0/4095891071 conn(0x7f51380fbc10 msgr2=0x7f51380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 -- 172.21.15.107:0/4095891071 shutdown_connections 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.859+0000 7f513ccd4640 1 -- 172.21.15.107:0/4095891071 wait complete. 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 Processor -- start 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 -- start start 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51381006a0 0x7f5138197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 0x7f513819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f513810d400 con 0x7f5138101890 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f513810d280 con 0x7f5138102250 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513ccd4640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f513810d580 con 0x7f51381006a0 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47372/0 (socket says 172.21.15.107:47372) 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 -- 172.21.15.107:0/384613759 learned_addr learned my addr 172.21.15.107:0/384613759 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f513659c640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51381006a0 0x7f5138197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5136d9d640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 0x7f513819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 -- 172.21.15.107:0/384613759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51381006a0 msgr2=0x7f5138197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51381006a0 0x7f5138197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 -- 172.21.15.107:0/384613759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 msgr2=0x7f513819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 0x7f513819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 -- 172.21.15.107:0/384613759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513819f750 con 0x7f5138101890 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f5135d9b640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f512800c890 tx=0x7f512800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.860+0000 7f51277fe640 1 -- 172.21.15.107:0/384613759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128004220 con 0x7f5138101890 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.861+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513819fa40 con 0x7f5138101890 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.861+0000 7f51277fe640 1 -- 172.21.15.107:0/384613759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51280043c0 con 0x7f5138101890 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.861+0000 7f51277fe640 1 -- 172.21.15.107:0/384613759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128004a00 con 0x7f5138101890 2024-04-24T23:44:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.861+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513819ff90 con 0x7f5138101890 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f51277fe640 1 -- 172.21.15.107:0/384613759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f51280079b0 con 0x7f5138101890 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 Processor -- start 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 -- start start 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b704860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b70c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56225b667f00 con 0x56225b704e30 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56225b610750 con 0x56225b709fa0 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.863+0000 7f513f9e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56225b663ec0 con 0x56225b704440 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47384/0 (socket says 172.21.15.107:47384) 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 -- 172.21.15.107:0/4078957734 learned_addr learned my addr 172.21.15.107:0/4078957734 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5135d9b640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b70c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f51277fe640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 0x7f51200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5136d9d640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b704860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 -- 172.21.15.107:0/4078957734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 msgr2=0x56225b704860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b704860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 -- 172.21.15.107:0/4078957734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 msgr2=0x56225b70c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b70c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 -- 172.21.15.107:0/4078957734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56225b37c2d0 con 0x56225b704e30 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5136d9d640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b704860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5136d9d640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 0x7f51200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f51277fe640 1 -- 172.21.15.107:0/384613759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51280cf180 con 0x7f5138101890 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f513659c640 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f512c0314e0 tx=0x7f512c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe674ed467616c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5124ff9640 1 -- 172.21.15.107:0/4078957734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c040070 con 0x56225b704e30 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5124ff9640 1 -- 172.21.15.107:0/4078957734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c005500 con 0x56225b704e30 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.864+0000 7f5124ff9640 1 -- 172.21.15.107:0/4078957734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c035040 con 0x56225b704e30 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- 172.21.15.107:0/4078957734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 msgr2=0x56225b709930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f5136d9d640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 0x7f51200a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x56225b705690 tx=0x7f5130004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f512c0314e0 tx=0x7f512c005c00 comp rx=0 tx=0).stop 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- 172.21.15.107:0/4078957734 shutdown_connections 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b70c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b709930 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4078957734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b704860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- 172.21.15.107:0/4078957734 >> 172.21.15.107:0/4078957734 conn(0x56225b6ffab0 msgr2=0x56225b6ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- 172.21.15.107:0/4078957734 shutdown_connections 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- 172.21.15.107:0/4078957734 wait complete. 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.865+0000 7f513f9e3740 1 -- start start 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 Processor -- start 2024-04-24T23:44:43.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b70e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b715d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56225b70cef0 con 0x56225b704e30 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56225b623cf0 con 0x56225b709fa0 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56225b70d070 con 0x56225b704440 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 10 client.0 _ll_register_callbacks cb 0x56225b6236e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47390/0 (socket says 172.21.15.107:47390) 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 -- 172.21.15.107:0/3761808896 learned_addr learned my addr 172.21.15.107:0/3761808896 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513659c640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b70e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5136d9d640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b715d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 -- 172.21.15.107:0/3761808896 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 msgr2=0x56225b70e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b70e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 -- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 msgr2=0x56225b715d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b715d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 -- 172.21.15.107:0/3761808896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56225b4466b0 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5136d9d640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b715d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f5135d9b640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5128097fa0 tx=0x7f51280053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b0ea6a571a0447 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f511dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512801d040 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5128084b60 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.866+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f512c005500 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f51280a4190 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 10 client.15312 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56225b718df0 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c005500 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f51280b4210 con 0x56225b704e30 2024-04-24T23:44:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 10 client.15312 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 10 client.15312 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 20 client.15312 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b71b4e0 0x56225b71c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 4 client.15312 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56225b675440 con 0x56225b71b4e0 2024-04-24T23:44:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.867+0000 7f513659c640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b71b4e0 0x56225b71c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f511dffb640 10 client.15312 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513659c640 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b71b4e0 0x56225b71c950 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b69045a5995a9acb server_cookie=5133dbfe82922cf5 in_seq=0 out_seq=0 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f511dffb640 1 -- 172.21.15.107:0/3761808896 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56225b675440 con 0x56225b71b4e0 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f511dffb640 10 client.15312 handle_command_reply: tid=0 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 client.15312 shutdown 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 msgr2=0x56225b70ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5128097fa0 tx=0x7f51280053a0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 shutdown_connections 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b71b4e0 0x56225b71c950 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b715d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b70ed70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 --2- 172.21.15.107:0/3761808896 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b70e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f511dffb640 0 client.15312 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 >> 172.21.15.107:0/3761808896 conn(0x56225b6ffab0 msgr2=0x56225b6feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 shutdown_connections 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.868+0000 7f513f9e3740 1 -- 172.21.15.107:0/3761808896 wait complete. 2024-04-24T23:44:43.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.869+0000 7f513f9e3740 20 client.15312 trim_cache size 0 max 16384 2024-04-24T23:44:43.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 Processor -- start 2024-04-24T23:44:43.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 -- start start 2024-04-24T23:44:43.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b6fef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b6fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56225b71ac90 con 0x56225b704e30 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56225b71b360 con 0x56225b709fa0 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f513f9e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56225b71ba30 con 0x56225b704440 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47406/0 (socket says 172.21.15.107:47406) 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.886+0000 7f5135d9b640 1 -- 172.21.15.107:0/4028251873 learned_addr learned my addr 172.21.15.107:0/4028251873 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5136d9d640 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b6fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 -- 172.21.15.107:0/4028251873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 msgr2=0x56225b6fef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b6fef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 -- 172.21.15.107:0/4028251873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 msgr2=0x56225b6fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b6fa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 -- 172.21.15.107:0/4028251873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56225b709510 con 0x56225b704e30 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5136d9d640 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b6fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f5135d9b640 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f512809b2d0 tx=0x7f5128005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66630596c47d5245 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f511ffff640 1 -- 172.21.15.107:0/4028251873 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512801d070 con 0x56225b704e30 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f511ffff640 1 -- 172.21.15.107:0/4028251873 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512809ad10 con 0x56225b704e30 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f511ffff640 1 -- 172.21.15.107:0/4028251873 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128084b60 con 0x56225b704e30 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f513f9e3740 1 -- 172.21.15.107:0/4028251873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 msgr2=0x56225b715d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.887+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f512809b2d0 tx=0x7f5128005300 comp rx=0 tx=0).stop 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- 172.21.15.107:0/4028251873 shutdown_connections 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b709fa0 0x56225b6fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704e30 0x56225b715d00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- 172.21.15.107:0/4028251873 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b704440 0x56225b6fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- 172.21.15.107:0/4028251873 >> 172.21.15.107:0/4028251873 conn(0x56225b6ffab0 msgr2=0x56225b6ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- 172.21.15.107:0/4028251873 shutdown_connections 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- 172.21.15.107:0/4028251873 wait complete. 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- start start 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 Processor -- start 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b704e30 0x56225b7123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b709fa0 0x56225b714fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56225b71c7d0 con 0x56225b704440 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56225b71c100 con 0x56225b704e30 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56225b71cea0 con 0x56225b709fa0 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513f9e3740 10 client.0 _ll_register_callbacks cb 0x7f51280057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47418/0 (socket says 172.21.15.107:47418) 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 -- 172.21.15.107:0/245017367 learned_addr learned my addr 172.21.15.107:0/245017367 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f5135d9b640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b704e30 0x56225b7123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 -- 172.21.15.107:0/245017367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b709fa0 msgr2=0x56225b714fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b709fa0 0x56225b714fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 -- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b704e30 msgr2=0x56225b7123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b704e30 0x56225b7123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.888+0000 7f513659c640 1 -- 172.21.15.107:0/245017367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56225b715620 con 0x56225b704440 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513659c640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f512c031bf0 tx=0x7f512c0315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3272ed1564c889c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c040050 con 0x56225b704440 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f512809ad10 con 0x56225b704440 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f512c005040 con 0x56225b704440 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f512c035040 con 0x56225b704440 2024-04-24T23:44:43.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 10 client.15318 fetch_fsmap learned FSMap version 28 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56225b781ab0 con 0x56225b704440 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512809ad10 con 0x56225b704440 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f512c0035f0 con 0x56225b704440 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 10 client.15318 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 10 client.15318 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 20 client.15318 populate_metadata read hostname 'smithi107' 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b786260 0x56225b784f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 4 client.15318 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "status"}) -- 0x56225b66f790 con 0x56225b786260 2024-04-24T23:44:43.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.889+0000 7f5135d9b640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b786260 0x56225b784f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f511dffb640 10 client.15318 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f5135d9b640 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b786260 0x56225b784f70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed0abb5f0308857 server_cookie=de7cbd3224e2fcce in_seq=0 out_seq=0 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f511dffb640 1 -- 172.21.15.107:0/245017367 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56225b66f790 con 0x56225b786260 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f511dffb640 10 client.15318 handle_command_reply: tid=0 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 client.15318 shutdown 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 msgr2=0x56225b711e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f512c031bf0 tx=0x7f512c0315b0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 shutdown_connections 2024-04-24T23:44:43.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56225b786260 0x56225b784f70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56225b709fa0 0x56225b714fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f511dffb640 0 client.15318 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56225b704e30 0x56225b7123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 --2- 172.21.15.107:0/245017367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56225b704440 0x56225b711e90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 >> 172.21.15.107:0/245017367 conn(0x56225b6ffab0 msgr2=0x56225b6fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 shutdown_connections 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 1 -- 172.21.15.107:0/245017367 wait complete. 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.890+0000 7f513f9e3740 20 client.15318 trim_cache size 0 max 16384 2024-04-24T23:44:43.900 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24506, 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 112.60903038399999, 2024-04-24T23:44:43.901 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:44:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-04-24T23:44:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:44:43.902 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 119.48354965 2024-04-24T23:44:43.902 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:44:43.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 msgr2=0x7f51200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 0x7f51200a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x56225b705690 tx=0x7f5130004a40 comp rx=0 tx=0).stop 2024-04-24T23:44:43.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 msgr2=0x7f5138198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:43.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f512800c890 tx=0x7f512800cd50 comp rx=0 tx=0).stop 2024-04-24T23:44:43.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 shutdown_connections 2024-04-24T23:44:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f51200a2c20 0x7f51200a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5138102250 0x7f513819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5138101890 0x7f5138198070 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 --2- 172.21.15.107:0/384613759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51381006a0 0x7f5138197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 >> 172.21.15.107:0/384613759 conn(0x7f51380fbc10 msgr2=0x7f5138104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:43.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 shutdown_connections 2024-04-24T23:44:43.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:43.891+0000 7f513ccd4640 1 -- 172.21.15.107:0/384613759 wait complete. 2024-04-24T23:44:43.907 INFO:tasks.cephadm:Setup complete, yielding 2024-04-24T23:44:43.908 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T23:44:43.919 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:44:43.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-24T23:44:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:44.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:44 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1813994124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:44.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:44 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1813994124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:44.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:44 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1813994124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:44:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:45 smithi107 ceph-mon[29118]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:45 smithi195 ceph-mon[33019]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:45.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:45 smithi170 ceph-mon[32714]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:44:45.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.944+0000 7ff2d6e15700 1 -- 172.21.15.107:0/2683936664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 msgr2=0x7ff2d01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.944+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/2683936664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d01095d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800b0a0 tx=0x7ff2c800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- 172.21.15.107:0/2683936664 shutdown_connections 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/2683936664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d01095d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/2683936664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0101180 0x7ff2d01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/2683936664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0102380 0x7ff2d01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- 172.21.15.107:0/2683936664 >> 172.21.15.107:0/2683936664 conn(0x7ff2d00fc720 msgr2=0x7ff2d00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- 172.21.15.107:0/2683936664 shutdown_connections 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- 172.21.15.107:0/2683936664 wait complete. 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 Processor -- start 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- start start 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0101180 0x7ff2d0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0102380 0x7ff2d0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010deb0 con 0x7ff2d0102d20 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010dd30 con 0x7ff2d0102380 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.945+0000 7ff2d6e15700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010e030 con 0x7ff2d0101180 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47422/0 (socket says 172.21.15.107:47422) 2024-04-24T23:44:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 -- 172.21.15.107:0/947306789 learned_addr learned my addr 172.21.15.107:0/947306789 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2cffff700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0102380 0x7ff2d0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d4bb1700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0101180 0x7ff2d0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 -- 172.21.15.107:0/947306789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0101180 msgr2=0x7ff2d0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0101180 0x7ff2d0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 -- 172.21.15.107:0/947306789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0102380 msgr2=0x7ff2d0198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0102380 0x7ff2d0198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 -- 172.21.15.107:0/947306789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d01a0060 con 0x7ff2d0102d20 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d53b2700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800d010 tx=0x7ff2c8004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8021070 con 0x7ff2d0102d20 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2c8007b80 con 0x7ff2d0102d20 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01a02f0 con 0x7ff2d0102d20 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.946+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c801d450 con 0x7ff2d0102d20 2024-04-24T23:44:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.947+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a07a0 con 0x7ff2d0102d20 2024-04-24T23:44:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.948+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d0068f20 con 0x7ff2d0102d20 2024-04-24T23:44:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.949+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff2c8004140 con 0x7ff2d0102d20 2024-04-24T23:44:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.949+0000 7ff2cdffb700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 0x7ff2c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.949+0000 7ff2d4bb1700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 0x7ff2c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.949+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff2c80d0610 con 0x7ff2d0102d20 2024-04-24T23:44:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.950+0000 7ff2d4bb1700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 0x7ff2c00a51f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff2d01021e0 tx=0x7ff2c4007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:45.954+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff2c809d140 con 0x7ff2d0102d20 2024-04-24T23:44:46.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.170+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7ff2d0062670 con 0x7ff2c00a2d30 2024-04-24T23:44:46.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.171+0000 7ff2cdffb700 1 -- 172.21.15.107:0/947306789 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ff2d0062670 con 0x7ff2c00a2d30 2024-04-24T23:44:46.170 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: cephadm 2024-04-24T23:44:46.171 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2024-04-24T23:44:46.171 INFO:teuthology.orchestra.run.smithi107.stdout:Paused: No 2024-04-24T23:44:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 msgr2=0x7ff2c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 0x7ff2c00a51f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff2d01021e0 tx=0x7ff2c4007400 comp rx=0 tx=0).stop 2024-04-24T23:44:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 msgr2=0x7ff2d019f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:46.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800d010 tx=0x7ff2c8004dd0 comp rx=0 tx=0).stop 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 shutdown_connections 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2c00a2d30 0x7ff2c00a51f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2d0102d20 0x7ff2d019f9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff2d0101180 0x7ff2d0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 --2- 172.21.15.107:0/947306789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d0102380 0x7ff2d0198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 >> 172.21.15.107:0/947306789 conn(0x7ff2d00fc720 msgr2=0x7ff2d0105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 shutdown_connections 2024-04-24T23:44:46.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:46.173+0000 7ff2d6e15700 1 -- 172.21.15.107:0/947306789 wait complete. 2024-04-24T23:44:46.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:46 smithi107 ceph-mon[29118]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:46.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:46 smithi107 ceph-mon[29118]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:46 smithi195 ceph-mon[33019]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:46 smithi195 ceph-mon[33019]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:46 smithi170 ceph-mon[32714]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:44:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:46 smithi170 ceph-mon[32714]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:46.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-24T23:44:47.208 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.022+0000 7f877c407700 1 -- 172.21.15.107:0/3909461099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 msgr2=0x7f8774100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.022+0000 7f877c407700 1 --2- 172.21.15.107:0/3909461099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774100a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f87700099c0 tx=0x7f8770009cd0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- 172.21.15.107:0/3909461099 shutdown_connections 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- 172.21.15.107:0/3909461099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8774104050 0x7f877410a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- 172.21.15.107:0/3909461099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774100a50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- 172.21.15.107:0/3909461099 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f87741036b0 0x7f8774103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- 172.21.15.107:0/3909461099 >> 172.21.15.107:0/3909461099 conn(0x7f87740fbf70 msgr2=0x7f87740fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- 172.21.15.107:0/3909461099 shutdown_connections 2024-04-24T23:44:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- 172.21.15.107:0/3909461099 wait complete. 2024-04-24T23:44:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 Processor -- start 2024-04-24T23:44:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- start start 2024-04-24T23:44:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87741036b0 0x7f8774101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8774104050 0x7f8774101ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f877410df40 con 0x7f8774100630 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f877410ddc0 con 0x7f87741036b0 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.023+0000 7f877c407700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f877410e0c0 con 0x7f8774104050 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32852/0 (socket says 172.21.15.107:32852) 2024-04-24T23:44:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 -- 172.21.15.107:0/3769268328 learned_addr learned my addr 172.21.15.107:0/3769268328 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f87799a2700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87741036b0 0x7f8774101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a9a4700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8774104050 0x7f8774101ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 -- 172.21.15.107:0/3769268328 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8774104050 msgr2=0x7f8774101ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8774104050 0x7f8774101ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 -- 172.21.15.107:0/3769268328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87741036b0 msgr2=0x7f8774101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87741036b0 0x7f8774101660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 -- 172.21.15.107:0/3769268328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8774102450 con 0x7f8774100630 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877a1a3700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f877000c010 tx=0x7f8770009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770027070 con 0x7f8774100630 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8770004150 con 0x7f8774100630 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87741a2650 con 0x7f8774100630 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877000f460 con 0x7f8774100630 2024-04-24T23:44:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.024+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87741a2b30 con 0x7f8774100630 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.025+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8774068f20 con 0x7f8774100630 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.026+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f877002a070 con 0x7f8774100630 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.027+0000 7f876b7fe700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 0x7f87600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.027+0000 7f87799a2700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 0x7f87600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.027+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f87700d0880 con 0x7f8774100630 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.028+0000 7f87799a2700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 0x7f87600a51f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f8764005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.031+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8770031070 con 0x7f8774100630 2024-04-24T23:44:49.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.247+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f8774062670 con 0x7f87600a2d30 2024-04-24T23:44:49.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.260+0000 7f876b7fe700 1 -- 172.21.15.107:0/3769268328 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f8774062670 con 0x7f87600a2d30 2024-04-24T23:44:49.261 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T23:44:49.261 INFO:teuthology.orchestra.run.smithi107.stdout:mds.a smithi107 running (2m) 32s ago 2m 15.7M - 19.0.0-2445-gd2c45d97 f68e37b807ef 19c771eaab50 2024-04-24T23:44:49.261 INFO:teuthology.orchestra.run.smithi107.stdout:mds.b smithi170 running (2m) 58s ago 2m 25.7M - 19.0.0-2445-gd2c45d97 f68e37b807ef 2322592b7fb6 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.c smithi195 running (2m) 58s ago 2m 16.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef c6fb4eaf38e4 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.d smithi107 running (2m) 32s ago 2m 13.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef 0649073ac5a6 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.e smithi170 running (2m) 58s ago 2m 18.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef ea811e32727d 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.f smithi195 running (2m) 58s ago 2m 18.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef ff46aeac811e 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.g smithi107 running (2m) 32s ago 2m 16.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef cf4a201e67e9 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.h smithi170 running (2m) 58s ago 2m 20.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef dfb83139b97f 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.i smithi195 running (2m) 58s ago 2m 20.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef ae6abb967957 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.j smithi107 running (2m) 32s ago 2m 13.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 17ce9952d998 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.k smithi170 running (2m) 58s ago 2m 18.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef bd204dd29889 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mds.l smithi195 running (2m) 58s ago 2m 16.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef b9c9e6ddeaba 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi107 *:9283,8765 running (9m) 32s ago 9m 514M - 19.0.0-2445-gd2c45d97 f68e37b807ef b0472117974e 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi170 *:8443,8765 running (7m) 58s ago 7m 452M - 19.0.0-2445-gd2c45d97 f68e37b807ef 081385a7b466 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.z smithi195 *:8443,8765 running (7m) 58s ago 7m 454M - 19.0.0-2445-gd2c45d97 f68e37b807ef cb54a4c4c63b 2024-04-24T23:44:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (9m) 32s ago 10m 50.4M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 14250468e610 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi170 running (8m) 58s ago 8m 39.7M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 0dbe7c606142 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi195 running (8m) 58s ago 8m 43.0M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 43ab3efaf319 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (7m) 32s ago 7m 46.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef ff810b2e0994 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi170 running (7m) 58s ago 7m 47.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef a8b1564bb500 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi195 running (6m) 58s ago 6m 59.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 19e9ab696fac 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (6m) 32s ago 6m 55.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 8734e2be7ab1 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi170 running (5m) 58s ago 5m 42.9M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 1710d2c12be7 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi195 running (5m) 58s ago 5m 45.3M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 928480ed146e 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi107 running (5m) 32s ago 5m 41.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 80164eb09401 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi170 running (4m) 58s ago 4m 58.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef e8cdd1c5d586 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 smithi195 running (4m) 58s ago 4m 56.1M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 6192bbb307df 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 smithi107 running (3m) 32s ago 3m 42.1M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 1b795252d77c 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 smithi170 running (3m) 58s ago 3m 43.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 4aad0317fbf6 2024-04-24T23:44:49.263 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 smithi195 running (2m) 58s ago 2m 42.0M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 483beb518f19 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 msgr2=0x7f87600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 0x7f87600a51f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f8764005eb0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 msgr2=0x7f8774103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f877000c010 tx=0x7f8770009f10 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 shutdown_connections 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f87600a2d30 0x7f87600a51f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8774104050 0x7f8774101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8774100630 0x7f8774103010 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 --2- 172.21.15.107:0/3769268328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87741036b0 0x7f8774101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 >> 172.21.15.107:0/3769268328 conn(0x7f87740fbf70 msgr2=0x7f8774106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.262+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 shutdown_connections 2024-04-24T23:44:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:49.263+0000 7f877c407700 1 -- 172.21.15.107:0/3769268328 wait complete. 2024-04-24T23:44:49.319 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:49 smithi195 ceph-mon[33019]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:49.319 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-04-24T23:44:49.319 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-24T23:44:49.319 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.319 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:49 smithi170 ceph-mon[32714]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-04-24T23:44:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-24T23:44:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:49 smithi107 ceph-mon[29118]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:44:49.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-04-24T23:44:49.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-24T23:44:49.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.383 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-24T23:44:49.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-24T23:44:50.225 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-04-24T23:44:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-24T23:44:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-24T23:44:50.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-24T23:44:50.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T23:44:50.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:50 smithi170 ceph-mon[32714]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T23:44:50.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:50 smithi107 ceph-mon[29118]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T23:44:50.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:50 smithi195 ceph-mon[33019]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:51 smithi170 ceph-mon[32714]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-24T23:44:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:51 smithi170 ceph-mon[32714]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:51 smithi170 ceph-mon[32714]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:51 smithi170 ceph-mon[32714]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:51 smithi107 ceph-mon[29118]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:51 smithi107 ceph-mon[29118]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:51 smithi107 ceph-mon[29118]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T23:44:51.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:51 smithi107 ceph-mon[29118]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T23:44:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:51 smithi195 ceph-mon[33019]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-24T23:44:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:51 smithi195 ceph-mon[33019]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-24T23:44:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:51 smithi195 ceph-mon[33019]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T23:44:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:51 smithi195 ceph-mon[33019]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 -- 172.21.15.107:0/359920381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103e20 msgr2=0x7fa80010a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 --2- 172.21.15.107:0/359920381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103e20 0x7fa80010a6d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800b0d0 tx=0x7fa7f800b3e0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 -- 172.21.15.107:0/359920381 shutdown_connections 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 --2- 172.21.15.107:0/359920381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103e20 0x7fa80010a6d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 --2- 172.21.15.107:0/359920381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800102280 0x7fa8001026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 --2- 172.21.15.107:0/359920381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800103480 0x7fa8001038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 -- 172.21.15.107:0/359920381 >> 172.21.15.107:0/359920381 conn(0x7fa8000fd820 msgr2=0x7fa8000ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 -- 172.21.15.107:0/359920381 shutdown_connections 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.894+0000 7fa806238700 1 -- 172.21.15.107:0/359920381 wait complete. 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 Processor -- start 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 -- start start 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800102280 0x7fa800198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 0x7fa80019f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80010deb0 con 0x7fa800103480 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa80010dd30 con 0x7fa800102280 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa806238700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa80010e030 con 0x7fa800103e20 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa7feffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.895+0000 7fa7feffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32864/0 (socket says 172.21.15.107:32864) 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 -- 172.21.15.107:0/3590673213 learned_addr learned my addr 172.21.15.107:0/3590673213 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7fffff700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 0x7fa80019f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7ff7fe700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800102280 0x7fa800198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 -- 172.21.15.107:0/3590673213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 msgr2=0x7fa80019f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 0x7fa80019f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 -- 172.21.15.107:0/3590673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800102280 msgr2=0x7fa800198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800102280 0x7fa800198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 -- 172.21.15.107:0/3590673213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8001a0070 con 0x7fa800103480 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7fffff700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 0x7fa80019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7feffd700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c9e0 tx=0x7fa7f400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4007ac0 con 0x7fa800103480 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8001a0360 con 0x7fa800103480 2024-04-24T23:44:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f4007c60 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4018710 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.896+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8001a08a0 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.897+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa800068f20 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.898+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa7f40188d0 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.899+0000 7fa7fcff9700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 0x7fa7e80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.899+0000 7fa7ff7fe700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 0x7fa7e80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.899+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7f40cf3c0 con 0x7fa800103480 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.899+0000 7fa7ff7fe700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 0x7fa7e80a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005950 tx=0x7fa7f00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:51.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:51.904+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa7f4014030 con 0x7fa800103480 2024-04-24T23:44:52.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.113+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa800062670 con 0x7fa7e80a2c20 2024-04-24T23:44:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.119+0000 7fa7fcff9700 1 -- 172.21.15.107:0/3590673213 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa800062670 con 0x7fa7e80a2c20 2024-04-24T23:44:52.118 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-24T23:44:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:mds.all 12/12 61s ago 2m smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi170=b;smithi170=e;smithi170=h;smithi170=k;smithi195=c;smithi195=f;smithi195=i;smithi195=l;count:12 2024-04-24T23:44:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 3/3 61s ago 8m smithi107=x;smithi170=y;smithi195=z;count:3 2024-04-24T23:44:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 61s ago 8m smithi107:172.21.15.107=a;smithi170:172.21.15.170=b;smithi195:172.21.15.195=c;count:3 2024-04-24T23:44:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:osd 12 61s ago - 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 msgr2=0x7fa7e80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 0x7fa7e80a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005950 tx=0x7fa7f00058e0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 msgr2=0x7fa800198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c9e0 tx=0x7fa7f400cda0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 shutdown_connections 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7e80a2c20 0x7fa7e80a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa800103e20 0x7fa80019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa800102280 0x7fa800198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 --2- 172.21.15.107:0/3590673213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa800103480 0x7fa800198950 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 >> 172.21.15.107:0/3590673213 conn(0x7fa8000fd820 msgr2=0x7fa8001066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:52.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 shutdown_connections 2024-04-24T23:44:52.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:52.121+0000 7fa806238700 1 -- 172.21.15.107:0/3590673213 wait complete. 2024-04-24T23:44:52.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-24T23:44:53.069 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:53.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:53 smithi107 ceph-mon[29118]: pgmap v355: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:53.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:53 smithi107 ceph-mon[29118]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:53 smithi170 ceph-mon[32714]: pgmap v355: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:53.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:53 smithi170 ceph-mon[32714]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:53 smithi195 ceph-mon[33019]: pgmap v355: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:53 smithi195 ceph-mon[33019]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 -- 172.21.15.107:0/664294077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0108040 msgr2=0x7fc5f010a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/664294077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0108040 0x7fc5f010a430 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800b0a0 tx=0x7fc5e800b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 -- 172.21.15.107:0/664294077 shutdown_connections 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/664294077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0108040 0x7fc5f010a430 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/664294077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0102de0 0x7fc5f01051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/664294077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0105710 0x7fc5f0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 -- 172.21.15.107:0/664294077 >> 172.21.15.107:0/664294077 conn(0x7fc5f00fc750 msgr2=0x7fc5f00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 -- 172.21.15.107:0/664294077 shutdown_connections 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.793+0000 7fc5f6cee700 1 -- 172.21.15.107:0/664294077 wait complete. 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 Processor -- start 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 -- start start 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0102de0 0x7fc5f0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 0x7fc5f019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f010dee0 con 0x7fc5f0105710 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f010dd60 con 0x7fc5f0108040 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f6cee700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f010e060 con 0x7fc5f0102de0 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:54.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32876/0 (socket says 172.21.15.107:32876) 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f4a8a700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0102de0 0x7fc5f0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 -- 172.21.15.107:0/1706390417 learned_addr learned my addr 172.21.15.107:0/1706390417 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f528b700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 0x7fc5f019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 -- 172.21.15.107:0/1706390417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0102de0 msgr2=0x7fc5f0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0102de0 0x7fc5f0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 -- 172.21.15.107:0/1706390417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 msgr2=0x7fc5f019fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 0x7fc5f019fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 -- 172.21.15.107:0/1706390417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f01a00d0 con 0x7fc5f0105710 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5f528b700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 0x7fc5f019fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.794+0000 7fc5effff700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00b560 tx=0x7fc5dc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.795+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc015020 con 0x7fc5f0105710 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.795+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f01a03c0 con 0x7fc5f0105710 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.795+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5dc011d10 con 0x7fc5f0105710 2024-04-24T23:44:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.795+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc010e50 con 0x7fc5f0105710 2024-04-24T23:44:54.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.795+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f01a0950 con 0x7fc5f0105710 2024-04-24T23:44:54.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.796+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f0068f20 con 0x7fc5f0105710 2024-04-24T23:44:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.797+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc5dc018020 con 0x7fc5f0105710 2024-04-24T23:44:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.797+0000 7fc5edffb700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 0x7fc5e00a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.798+0000 7fc5f4a8a700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 0x7fc5e00a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.798+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc5dc0ceb90 con 0x7fc5f0105710 2024-04-24T23:44:54.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.798+0000 7fc5f4a8a700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 0x7fc5e00a5280 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4005950 tx=0x7fc5e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:54.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:54.802+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5dc01f020 con 0x7fc5f0105710 2024-04-24T23:44:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.005+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc5f0062670 con 0x7fc5e00a2dc0 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.006+0000 7fc5edffb700 1 -- 172.21.15.107:0/1706390417 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fc5f0062670 con 0x7fc5e00a2dc0 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stdout:HOST ADDR LABELS STATUS 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 172.21.15.107 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stdout:smithi170 172.21.15.170 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 172.21.15.195 2024-04-24T23:44:55.006 INFO:teuthology.orchestra.run.smithi107.stdout:3 hosts in cluster 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 msgr2=0x7fc5e00a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 0x7fc5e00a5280 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4005950 tx=0x7fc5e400b410 comp rx=0 tx=0).stop 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 msgr2=0x7fc5f01989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00b560 tx=0x7fc5dc00b870 comp rx=0 tx=0).stop 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 shutdown_connections 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc5e00a2dc0 0x7fc5e00a5280 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f0108040 0x7fc5f019fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc5f0102de0 0x7fc5f0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 --2- 172.21.15.107:0/1706390417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5f0105710 0x7fc5f01989b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 >> 172.21.15.107:0/1706390417 conn(0x7fc5f00fc750 msgr2=0x7fc5f0103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 shutdown_connections 2024-04-24T23:44:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:55.008+0000 7fc5f6cee700 1 -- 172.21.15.107:0/1706390417 wait complete. 2024-04-24T23:44:55.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:55 smithi170 ceph-mon[32714]: pgmap v356: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-24T23:44:55.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:55 smithi107 ceph-mon[29118]: pgmap v356: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-24T23:44:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:55 smithi195 ceph-mon[33019]: pgmap v356: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-04-24T23:44:55.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-24T23:44:55.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:56.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:56 smithi170 ceph-mon[32714]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:56.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:56 smithi107 ceph-mon[29118]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:56.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:56 smithi195 ceph-mon[33019]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:57.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:57 smithi170 ceph-mon[32714]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:57.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:57 smithi170 ceph-mon[32714]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-24T23:44:57.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:57 smithi170 ceph-mon[32714]: Cluster is now healthy 2024-04-24T23:44:57.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:57 smithi107 ceph-mon[29118]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:57.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:57 smithi107 ceph-mon[29118]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-24T23:44:57.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:57 smithi107 ceph-mon[29118]: Cluster is now healthy 2024-04-24T23:44:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:57 smithi195 ceph-mon[33019]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-24T23:44:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:57 smithi195 ceph-mon[33019]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-24T23:44:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:57 smithi195 ceph-mon[33019]: Cluster is now healthy 2024-04-24T23:44:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 -- 172.21.15.107:0/2442060422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda38103f20 msgr2=0x7fda3810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 --2- 172.21.15.107:0/2442060422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda38103f20 0x7fda3810a7d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fda3000b0a0 tx=0x7fda3000b3b0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 -- 172.21.15.107:0/2442060422 shutdown_connections 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 --2- 172.21.15.107:0/2442060422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda38103f20 0x7fda3810a7d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 --2- 172.21.15.107:0/2442060422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda381005a0 0x7fda381009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 --2- 172.21.15.107:0/2442060422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103580 0x7fda381039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.723+0000 7fda3f488700 1 -- 172.21.15.107:0/2442060422 >> 172.21.15.107:0/2442060422 conn(0x7fda380fbf00 msgr2=0x7fda380fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- 172.21.15.107:0/2442060422 shutdown_connections 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- 172.21.15.107:0/2442060422 wait complete. 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 Processor -- start 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- start start 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 0x7fda38198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103f20 0x7fda3819f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda3810deb0 con 0x7fda381005a0 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fda3810dd30 con 0x7fda38103580 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3f488700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fda3810e030 con 0x7fda38103f20 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3d224700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3d224700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37922/0 (socket says 172.21.15.107:37922) 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.724+0000 7fda3d224700 1 -- 172.21.15.107:0/1494243948 learned_addr learned my addr 172.21.15.107:0/1494243948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:44:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3ca23700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 0x7fda38198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3da25700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103f20 0x7fda3819f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 -- 172.21.15.107:0/1494243948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103f20 msgr2=0x7fda3819f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103f20 0x7fda3819f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 -- 172.21.15.107:0/1494243948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 msgr2=0x7fda38198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 0x7fda38198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 -- 172.21.15.107:0/1494243948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda381a0060 con 0x7fda381005a0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3ca23700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 0x7fda38198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3d224700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fda3400e9a0 tx=0x7fda3400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3400cbd0 con 0x7fda381005a0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda381a0350 con 0x7fda381005a0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda34004500 con 0x7fda381005a0 2024-04-24T23:44:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34010430 con 0x7fda381005a0 2024-04-24T23:44:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.725+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda381a08e0 con 0x7fda381005a0 2024-04-24T23:44:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.726+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda38068f20 con 0x7fda381005a0 2024-04-24T23:44:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.727+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fda34003f40 con 0x7fda381005a0 2024-04-24T23:44:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.728+0000 7fda2e7fc700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 0x7fda240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:44:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.728+0000 7fda3ca23700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 0x7fda240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:44:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.728+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fda340cf820 con 0x7fda381005a0 2024-04-24T23:44:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.728+0000 7fda3ca23700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 0x7fda240a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fda2800bcf0 tx=0x7fda2800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:44:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.731+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda3401f340 con 0x7fda381005a0 2024-04-24T23:44:57.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.936+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fda38062670 con 0x7fda240a2c20 2024-04-24T23:44:57.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.937+0000 7fda2e7fc700 1 -- 172.21.15.107:0/1494243948 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fda38062670 con 0x7fda240a2c20 2024-04-24T23:44:57.937 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-24T23:44:57.937 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-24T23:44:57.937 INFO:teuthology.orchestra.run.smithi107.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-24T23:44:57.937 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300FS480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-24T23:44:57.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 msgr2=0x7fda240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 0x7fda240a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fda2800bcf0 tx=0x7fda2800b2c0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 msgr2=0x7fda38198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fda3400e9a0 tx=0x7fda3400ed60 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 shutdown_connections 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda240a2c20 0x7fda240a50e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda38103f20 0x7fda3819f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda381005a0 0x7fda38198400 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 --2- 172.21.15.107:0/1494243948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda38103580 0x7fda38198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.939+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 >> 172.21.15.107:0/1494243948 conn(0x7fda380fbf00 msgr2=0x7fda381067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.940+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 shutdown_connections 2024-04-24T23:44:57.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:44:57.940+0000 7fda3f488700 1 -- 172.21.15.107:0/1494243948 wait complete. 2024-04-24T23:44:58.677 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T23:44:58.687 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:44:58.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-24T23:44:58.925 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:44:59.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:59 smithi107 ceph-mon[29118]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:59.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:44:59 smithi107 ceph-mon[29118]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 457 B/s wr, 11 op/s 2024-04-24T23:44:59.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:59 smithi170 ceph-mon[32714]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:59.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:44:59 smithi170 ceph-mon[32714]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 457 B/s wr, 11 op/s 2024-04-24T23:44:59.364 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:59 smithi195 ceph-mon[33019]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T23:44:59.364 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:44:59 smithi195 ceph-mon[33019]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 457 B/s wr, 11 op/s 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.597+0000 7fe0cc380700 1 -- 172.21.15.107:0/1517901726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4108000 msgr2=0x7fe0c410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.597+0000 7fe0cc380700 1 --2- 172.21.15.107:0/1517901726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4108000 0x7fe0c410a3f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00b0a0 tx=0x7fe0bc00b3b0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- 172.21.15.107:0/1517901726 shutdown_connections 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- 172.21.15.107:0/1517901726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4108000 0x7fe0c410a3f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- 172.21.15.107:0/1517901726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4102da0 0x7fe0c4105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- 172.21.15.107:0/1517901726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- 172.21.15.107:0/1517901726 >> 172.21.15.107:0/1517901726 conn(0x7fe0c40fc730 msgr2=0x7fe0c40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:00.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- 172.21.15.107:0/1517901726 shutdown_connections 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- 172.21.15.107:0/1517901726 wait complete. 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 Processor -- start 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- start start 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 0x7fe0c419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c410dec0 con 0x7fe0c4102da0 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c410dd40 con 0x7fe0c4108000 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.598+0000 7fe0cc380700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c410e040 con 0x7fe0c41056d0 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37942/0 (socket says 172.21.15.107:37942) 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 -- 172.21.15.107:0/335335025 learned_addr learned my addr 172.21.15.107:0/335335025 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:00.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca91d700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 0x7fe0c419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 -- 172.21.15.107:0/335335025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 msgr2=0x7fe0c4198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0c991b700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 -- 172.21.15.107:0/335335025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 msgr2=0x7fe0c419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 0x7fe0c419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 -- 172.21.15.107:0/335335025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c41a0110 con 0x7fe0c4102da0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0c991b700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca91d700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 0x7fe0c419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0ca11c700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000b560 tx=0x7fe0c000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0015020 con 0x7fe0c4102da0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c0011d10 con 0x7fe0c4102da0 2024-04-24T23:45:00.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c41a0400 con 0x7fe0c4102da0 2024-04-24T23:45:00.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c000f2e0 con 0x7fe0c4102da0 2024-04-24T23:45:00.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.599+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c41a0940 con 0x7fe0c4102da0 2024-04-24T23:45:00.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.600+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c4068f20 con 0x7fe0c4102da0 2024-04-24T23:45:00.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.601+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe0c0018020 con 0x7fe0c4102da0 2024-04-24T23:45:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.602+0000 7fe0bb7fe700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 0x7fe0b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.602+0000 7fe0c991b700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 0x7fe0b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.602+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe0c00cec60 con 0x7fe0c4102da0 2024-04-24T23:45:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.602+0000 7fe0c991b700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 0x7fe0b00a50e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4005fd0 tx=0x7fe0b4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:00.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.606+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe0c001f020 con 0x7fe0c4102da0 2024-04-24T23:45:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.845+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fe0c402db30 con 0x7fe0c4102da0 2024-04-24T23:45:00.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.845+0000 7fe0bb7fe700 1 -- 172.21.15.107:0/335335025 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3108 (secure 0 0 0) 0x7fe0c009b7f0 con 0x7fe0c4102da0 2024-04-24T23:45:00.845 INFO:teuthology.orchestra.run.smithi107.stdout:e28 2024-04-24T23:45:00.845 INFO:teuthology.orchestra.run.smithi107.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T23:45:00.845 INFO:teuthology.orchestra.run.smithi107.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:legacy client fscid: 1 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:Filesystem 'cephfs' (1) 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:fs_name cephfs 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 28 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-04-24T23:42:50.405146+0000 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-04-24T23:43:19.093489+0000 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:tableserver 0 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:root 0 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:session_timeout 300 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:session_autoclose 300 2024-04-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi107.stdout:max_file_size 1099511627776 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:max_xattr_size 65536 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:required_client_features {} 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure 0 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure_osd_epoch 0 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:max_mds 5 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:in 0,1,2,3,4 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:up {0=24506,1=24493,2=24464,3=24463,4=24451} 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:failed 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:damaged 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:stopped 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:data_pools [3,4] 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:metadata_pool 2 2024-04-24T23:45:00.847 INFO:teuthology.orchestra.run.smithi107.stdout:inline_data disabled 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:balancer 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:bal_rank_mask -1 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:standby_count_wanted 0 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:qdb_cluster leader: 24506 members: 24464,24506,24463,24493,24451 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b{0:24506} state up:active seq 3 addr [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.e{0:24431} state up:standby-replay seq 1 addr [v2:172.21.15.170:6834/3179503985,v1:172.21.15.170:6835/3179503985] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.i{1:24493} state up:active seq 11 addr [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.k{1:24419} state up:standby-replay seq 1 addr [v2:172.21.15.170:6832/3444431424,v1:172.21.15.170:6833/3444431424] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.h{2:24464} state up:active seq 12 addr [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.l{2:24406} state up:standby-replay seq 1 addr [v2:172.21.15.195:6832/1442081594,v1:172.21.15.195:6833/1442081594] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c{3:24463} state up:active seq 13 addr [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a{3:14646} state up:standby-replay seq 1 addr [v2:172.21.15.107:6840/1612113576,v1:172.21.15.107:6841/1612113576] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.f{4:24451} state up:active seq 14 addr [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.g{4:14634} state up:standby-replay seq 1 addr [v2:172.21.15.107:6838/527084969,v1:172.21.15.107:6839/527084969] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout:Standby daemons: 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.107:6834/3058072142,v1:172.21.15.107:6835/3058072142] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.107:6836/3538832535,v1:172.21.15.107:6837/3538832535] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 msgr2=0x7fe0b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 0x7fe0b00a50e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4005fd0 tx=0x7fe0b4005ee0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 msgr2=0x7fe0c4198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000b560 tx=0x7fe0c000b920 comp rx=0 tx=0).stop 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 shutdown_connections 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe0b00a2c20 0x7fe0b00a50e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0c4108000 0x7fe0c419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0c4102da0 0x7fe0c4198500 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 --2- 172.21.15.107:0/335335025 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe0c41056d0 0x7fe0c4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:00.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 >> 172.21.15.107:0/335335025 conn(0x7fe0c40fc730 msgr2=0x7fe0c41036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:00.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 shutdown_connections 2024-04-24T23:45:00.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:00.847+0000 7fe0cc380700 1 -- 172.21.15.107:0/335335025 wait complete. 2024-04-24T23:45:00.851 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:45:01.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:01 smithi107 ceph-mon[29118]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-24T23:45:01.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:01 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/335335025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T23:45:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:01 smithi195 ceph-mon[33019]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-24T23:45:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:01 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/335335025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T23:45:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:01 smithi170 ceph-mon[32714]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-24T23:45:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:01 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/335335025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T23:45:01.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-24T23:45:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:45:03.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:03 smithi107 ceph-mon[29118]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 257 B/s wr, 10 op/s 2024-04-24T23:45:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:03 smithi195 ceph-mon[33019]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 257 B/s wr, 10 op/s 2024-04-24T23:45:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:03 smithi170 ceph-mon[32714]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 257 B/s wr, 10 op/s 2024-04-24T23:45:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/38617178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18103f90 msgr2=0x7fcb1810a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:03.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/38617178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18103f90 0x7fcb1810a840 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcb1000b0a0 tx=0x7fcb1000b3b0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/38617178 shutdown_connections 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/38617178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18103f90 0x7fcb1810a840 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/38617178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18100610 0x7fcb18100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/38617178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 0x7fcb18103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/38617178 >> 172.21.15.107:0/38617178 conn(0x7fcb180fbf70 msgr2=0x7fcb180fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/38617178 shutdown_connections 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/38617178 wait complete. 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.592+0000 7fcb1e0c0700 1 Processor -- start 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 -- start start 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 0x7fcb1819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18103f90 0x7fcb181a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:03.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1810e020 con 0x7fcb18100610 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1810dea0 con 0x7fcb181035f0 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb1e0c0700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1810e1a0 con 0x7fcb18103f90 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37950/0 (socket says 172.21.15.107:37950) 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 -- 172.21.15.107:0/3535785529 learned_addr learned my addr 172.21.15.107:0/3535785529 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb16ffd700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 0x7fcb1819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb17fff700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18103f90 0x7fcb181a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 -- 172.21.15.107:0/3535785529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18103f90 msgr2=0x7fcb181a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18103f90 0x7fcb181a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 -- 172.21.15.107:0/3535785529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 msgr2=0x7fcb1819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 0x7fcb1819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.593+0000 7fcb177fe700 1 -- 172.21.15.107:0/3535785529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb181a2450 con 0x7fcb18100610 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb177fe700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800b530 tx=0x7fcb0800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:03.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08015020 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb181a2740 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb08011d10 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb181a2cd0 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.594+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08010e30 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.595+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb18068f20 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.596+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fcb08018020 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.597+0000 7fcb14ff9700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 0x7fcb000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.597+0000 7fcb16ffd700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 0x7fcb000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.597+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb080cf130 con 0x7fcb18100610 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.598+0000 7fcb16ffd700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 0x7fcb000a50e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c006fd0 tx=0x7fcb0c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:03.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.601+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb0801f020 con 0x7fcb18100610 2024-04-24T23:45:03.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.799+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fcb181a2f90 con 0x7fcb18100610 2024-04-24T23:45:03.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.800+0000 7fcb14ff9700 1 -- 172.21.15.107:0/3535785529 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5733 (secure 0 0 0) 0x7fcb181a2f90 con 0x7fcb18100610 2024-04-24T23:45:03.799 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 93 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-04-24T23:34:45.996336+0000 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-04-24T23:44:50.063628+0000 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 27 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client luminous 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release squid 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-04-24T23:45:03.800 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 12 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6802/1988153607,v1:172.21.15.107:6803/1988153607] [v2:172.21.15.107:6804/1988153607,v1:172.21.15.107:6805/1988153607] exists,up 7adc01bb-2472-44e3-abc7-5ef350338376 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/80173128,v1:172.21.15.170:6801/80173128] [v2:172.21.15.170:6802/80173128,v1:172.21.15.170:6803/80173128] exists,up 6f9087e7-4eaa-4516-b08f-3eef8dbae2d5 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6800/3967620404,v1:172.21.15.195:6801/3967620404] [v2:172.21.15.195:6802/3967620404,v1:172.21.15.195:6803/3967620404] exists,up 010d750a-e582-46e0-ac12-95fedac8d734 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6810/3900016553,v1:172.21.15.107:6811/3900016553] [v2:172.21.15.107:6812/3900016553,v1:172.21.15.107:6813/3900016553] exists,up 14800c24-129e-4081-bb09-cbef9af89c78 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/2310043947,v1:172.21.15.170:6809/2310043947] [v2:172.21.15.170:6810/2310043947,v1:172.21.15.170:6811/2310043947] exists,up 7310213f-3a15-436a-a0bc-38c005a7aeb8 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6808/1022606297,v1:172.21.15.195:6809/1022606297] [v2:172.21.15.195:6810/1022606297,v1:172.21.15.195:6811/1022606297] exists,up e45f9071-7234-40da-87e6-04473b1a0b07 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6818/3398193587,v1:172.21.15.107:6819/3398193587] [v2:172.21.15.107:6820/3398193587,v1:172.21.15.107:6821/3398193587] exists,up 1033c741-8643-41f0-b5e3-a7dfbb4d47d0 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/2352399844,v1:172.21.15.170:6817/2352399844] [v2:172.21.15.170:6818/2352399844,v1:172.21.15.170:6819/2352399844] exists,up 42a4f20d-28a4-4910-9850-ebd9c9919974 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6816/2912271513,v1:172.21.15.195:6817/2912271513] [v2:172.21.15.195:6818/2912271513,v1:172.21.15.195:6819/2912271513] exists,up e9844dda-7739-49ea-9548-685903258f80 2024-04-24T23:45:03.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6826/150997445,v1:172.21.15.107:6827/150997445] [v2:172.21.15.107:6828/150997445,v1:172.21.15.107:6829/150997445] exists,up 98528d16-0e91-4f6c-99ed-71e534aa6551 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/1507442612,v1:172.21.15.170:6825/1507442612] [v2:172.21.15.170:6826/1507442612,v1:172.21.15.170:6827/1507442612] exists,up eef6b39d-cb02-4439-9562-fb2ee00ed40d 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6824/2390923874,v1:172.21.15.195:6825/2390923874] [v2:172.21.15.195:6826/2390923874,v1:172.21.15.195:6827/2390923874] exists,up fc99798e-5b23-45a7-9bf3-40c43d5d092e 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.5 [8,11] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.6 [8,11] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.16 [2,4] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.1d [2,1] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.11 [4,6] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.18 [3,9] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.1a [5,7] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.21 [4,8] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.24 [4,9] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.37 [1,9] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.a [5,10] 2024-04-24T23:45:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.e [9,11] 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.f [9,8] 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.19 [9,8] 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/1738433151 expires 2024-04-25T23:35:47.891427+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/1738433151 expires 2024-04-25T23:35:47.891427+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4054350187 expires 2024-04-25T23:35:47.891427+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/1957718163 expires 2024-04-25T23:35:21.924087+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4085406700 expires 2024-04-25T23:35:47.891427+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/526392411 expires 2024-04-25T23:35:21.924087+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1308114869 expires 2024-04-25T23:35:21.924087+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2998673885 expires 2024-04-25T23:35:21.924087+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/1957718163 expires 2024-04-25T23:35:21.924087+0000 2024-04-24T23:45:03.803 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2063300390 expires 2024-04-25T23:35:47.891427+0000 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 msgr2=0x7fcb000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 0x7fcb000a50e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c006fd0 tx=0x7fcb0c008040 comp rx=0 tx=0).stop 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 msgr2=0x7fcb1819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800b530 tx=0x7fcb0800b8f0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 shutdown_connections 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcb000a2c20 0x7fcb000a50e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcb18103f90 0x7fcb181a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb18100610 0x7fcb1819a7f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 --2- 172.21.15.107:0/3535785529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb181035f0 0x7fcb1819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 >> 172.21.15.107:0/3535785529 conn(0x7fcb180fbf70 msgr2=0x7fcb181013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 shutdown_connections 2024-04-24T23:45:03.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:03.802+0000 7fcb1e0c0700 1 -- 172.21.15.107:0/3535785529 wait complete. 2024-04-24T23:45:04.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:04 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3535785529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T23:45:04.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:04 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3535785529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T23:45:04.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:04 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3535785529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T23:45:04.631 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-24T23:45:04.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/3328208375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 msgr2=0x7ffa180ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/3328208375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa180ffe80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009930 tx=0x7ffa0802a420 comp rx=0 tx=0).stop 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/3328208375 shutdown_connections 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/3328208375 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa181033f0 0x7ffa18109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/3328208375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa18102a30 0x7ffa18102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/3328208375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa180ffe80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/3328208375 >> 172.21.15.107:0/3328208375 conn(0x7ffa180fb3b0 msgr2=0x7ffa180fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/3328208375 shutdown_connections 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.748+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/3328208375 wait complete. 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 Processor -- start 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 -- start start 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa18102a30 0x7ffa18197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa181033f0 0x7ffa1819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810d490 con 0x7ffa180ffa60 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810d310 con 0x7ffa181033f0 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa1d5c1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810d610 con 0x7ffa18102a30 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37956/0 (socket says 172.21.15.107:37956) 2024-04-24T23:45:04.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 -- 172.21.15.107:0/1165180984 learned_addr learned my addr 172.21.15.107:0/1165180984 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa167fc640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa18102a30 0x7ffa18197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 -- 172.21.15.107:0/1165180984 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa18102a30 msgr2=0x7ffa18197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa18102a30 0x7ffa18197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 -- 172.21.15.107:0/1165180984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa181033f0 msgr2=0x7ffa1819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.749+0000 7ffa16ffd640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa181033f0 0x7ffa1819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ffa16ffd640 1 -- 172.21.15.107:0/1165180984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1819f6d0 con 0x7ffa180ffa60 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ffa16ffd640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffa08031440 tx=0x7ffa080045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0803f030 con 0x7ffa180ffa60 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1819f960 con 0x7ffa180ffa60 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa08003ce0 con 0x7ffa180ffa60 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08035070 con 0x7ffa180ffa60 2024-04-24T23:45:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.750+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1819fed0 con 0x7ffa180ffa60 2024-04-24T23:45:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.751+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa18108b00 con 0x7ffa180ffa60 2024-04-24T23:45:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.752+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ffa08042030 con 0x7ffa180ffa60 2024-04-24T23:45:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.752+0000 7ff9fffff640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 0x7ffa000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.752+0000 7ffa167fc640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 0x7ffa000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:04.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.753+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffa080efaf0 con 0x7ffa180ffa60 2024-04-24T23:45:04.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.753+0000 7ffa167fc640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 0x7ffa000a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005e10 tx=0x7ffa0c005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:04.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.757+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa080bc680 con 0x7ffa180ffa60 2024-04-24T23:45:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.934+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa18062410 con 0x7ffa180ffa60 2024-04-24T23:45:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.936+0000 7ff9fffff640 1 -- 172.21.15.107:0/1165180984 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7ffa0803b460 con 0x7ffa180ffa60 2024-04-24T23:45:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:04.936 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 msgr2=0x7ffa000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 0x7ffa000a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005e10 tx=0x7ffa0c005da0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 msgr2=0x7ffa18197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffa08031440 tx=0x7ffa080045e0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 shutdown_connections 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa000a2b90 0x7ffa000a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa181033f0 0x7ffa1819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa18102a30 0x7ffa18197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 --2- 172.21.15.107:0/1165180984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa180ffa60 0x7ffa18197ab0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 >> 172.21.15.107:0/1165180984 conn(0x7ffa180fb3b0 msgr2=0x7ffa18105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 shutdown_connections 2024-04-24T23:45:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:04.938+0000 7ffa1d5c1640 1 -- 172.21.15.107:0/1165180984 wait complete. 2024-04-24T23:45:04.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:45:04.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:45:05.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 -- 172.21.15.107:0/3221736456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50103340 msgr2=0x7faf50109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 --2- 172.21.15.107:0/3221736456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50103340 0x7faf50109bd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7faf48009ca0 tx=0x7faf4802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 -- 172.21.15.107:0/3221736456 shutdown_connections 2024-04-24T23:45:05.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 --2- 172.21.15.107:0/3221736456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50103340 0x7faf50109bd0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 --2- 172.21.15.107:0/3221736456 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50102980 0x7faf50102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 --2- 172.21.15.107:0/3221736456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50101790 0x7faf50101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 -- 172.21.15.107:0/3221736456 >> 172.21.15.107:0/3221736456 conn(0x7faf500fcd20 msgr2=0x7faf500ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 -- 172.21.15.107:0/3221736456 shutdown_connections 2024-04-24T23:45:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.051+0000 7faf57015640 1 -- 172.21.15.107:0/3221736456 wait complete. 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 Processor -- start 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 -- start start 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 0x7faf50197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50103340 0x7faf5019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf5010d470 con 0x7faf50102980 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faf5010d2f0 con 0x7faf50103340 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf57015640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7faf5010d5f0 con 0x7faf50101790 2024-04-24T23:45:05.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37972/0 (socket says 172.21.15.107:37972) 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf4ffff640 1 -- 172.21.15.107:0/3380412511 learned_addr learned my addr 172.21.15.107:0/3380412511 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf54db1640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 0x7faf50197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.052+0000 7faf4ffff640 1 -- 172.21.15.107:0/3380412511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 msgr2=0x7faf50197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4ffff640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 0x7faf50197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf555b2640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50103340 0x7faf5019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4ffff640 1 -- 172.21.15.107:0/3380412511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50103340 msgr2=0x7faf5019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4ffff640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50103340 0x7faf5019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4ffff640 1 -- 172.21.15.107:0/3380412511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5019f700 con 0x7faf50102980 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf54db1640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 0x7faf50197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4ffff640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00b4a0 tx=0x7faf3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c014070 con 0x7faf50102980 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf3c010af0 con 0x7faf50102980 2024-04-24T23:45:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5019f9f0 con 0x7faf50102980 2024-04-24T23:45:05.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c017d10 con 0x7faf50102980 2024-04-24T23:45:05.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.053+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5019fef0 con 0x7faf50102980 2024-04-24T23:45:05.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.054+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf501089e0 con 0x7faf50102980 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.055+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7faf3c017070 con 0x7faf50102980 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.055+0000 7faf4dffb640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 0x7faf400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.056+0000 7faf54db1640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 0x7faf400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.056+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faf3c0ce900 con 0x7faf50102980 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.056+0000 7faf54db1640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 0x7faf400a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf501027e0 tx=0x7faf44005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.060+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf3c09b490 con 0x7faf50102980 2024-04-24T23:45:05.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.235+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf50062340 con 0x7faf50102980 2024-04-24T23:45:05.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.236+0000 7faf4dffb640 1 -- 172.21.15.107:0/3380412511 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7faf3c09ad90 con 0x7faf50102980 2024-04-24T23:45:05.236 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 msgr2=0x7faf400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 0x7faf400a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf501027e0 tx=0x7faf44005db0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 msgr2=0x7faf50198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00b4a0 tx=0x7faf3c00b960 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 shutdown_connections 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7faf400a2b90 0x7faf400a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf50103340 0x7faf5019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf50102980 0x7faf50198020 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 --2- 172.21.15.107:0/3380412511 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faf50101790 0x7faf50197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 >> 172.21.15.107:0/3380412511 conn(0x7faf500fcd20 msgr2=0x7faf50105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 shutdown_connections 2024-04-24T23:45:05.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.238+0000 7faf57015640 1 -- 172.21.15.107:0/3380412511 wait complete. 2024-04-24T23:45:05.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:05 smithi107 ceph-mon[29118]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:05.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1165180984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:05.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- 172.21.15.107:0/767657905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 msgr2=0x7fd5b4102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 --2- 172.21.15.107:0/767657905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4102e00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009900 tx=0x7fd59c02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- 172.21.15.107:0/767657905 shutdown_connections 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 --2- 172.21.15.107:0/767657905 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4103340 0x7fd5b4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 --2- 172.21.15.107:0/767657905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4102e00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 --2- 172.21.15.107:0/767657905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4101790 0x7fd5b4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- 172.21.15.107:0/767657905 >> 172.21.15.107:0/767657905 conn(0x7fd5b40fcd20 msgr2=0x7fd5b40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- 172.21.15.107:0/767657905 shutdown_connections 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- 172.21.15.107:0/767657905 wait complete. 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 Processor -- start 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.350+0000 7fd5bad70640 1 -- start start 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 0x7fd5b4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4103340 0x7fd5b419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b410d3f0 con 0x7fd5b4102980 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b410d270 con 0x7fd5b4103340 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b410d570 con 0x7fd5b4101790 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37988/0 (socket says 172.21.15.107:37988) 2024-04-24T23:45:05.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 -- 172.21.15.107:0/2382541370 learned_addr learned my addr 172.21.15.107:0/2382541370 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b8b0c640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 0x7fd5b4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b930d640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4103340 0x7fd5b419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 -- 172.21.15.107:0/2382541370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 msgr2=0x7fd5b4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 0x7fd5b4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 -- 172.21.15.107:0/2382541370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4103340 msgr2=0x7fd5b419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4103340 0x7fd5b419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 -- 172.21.15.107:0/2382541370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b419f6e0 con 0x7fd5b4102980 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b8b0c640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 0x7fd5b4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b3fff640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009230 tx=0x7fd59c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c03f030 con 0x7fd5b4102980 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59c003ce0 con 0x7fd5b4102980 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b419f970 con 0x7fd5b4102980 2024-04-24T23:45:05.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.351+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c035070 con 0x7fd5b4102980 2024-04-24T23:45:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.352+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b419fe60 con 0x7fd5b4102980 2024-04-24T23:45:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.352+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b4108bd0 con 0x7fd5b4102980 2024-04-24T23:45:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.354+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd59c042030 con 0x7fd5b4102980 2024-04-24T23:45:05.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.354+0000 7fd5b1ffb640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 0x7fd5a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.354+0000 7fd5b8b0c640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 0x7fd5a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.354+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd59c0efee0 con 0x7fd5b4102980 2024-04-24T23:45:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.355+0000 7fd5b8b0c640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 0x7fd5a00a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd5b41027e0 tx=0x7fd5ac009400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.358+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd59c0bca70 con 0x7fd5b4102980 2024-04-24T23:45:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.532+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5b402d300 con 0x7fd5b4102980 2024-04-24T23:45:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.532+0000 7fd5b1ffb640 1 -- 172.21.15.107:0/2382541370 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fd59c0bc370 con 0x7fd5b4102980 2024-04-24T23:45:05.532 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 msgr2=0x7fd5a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 0x7fd5a00a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd5b41027e0 tx=0x7fd5ac009400 comp rx=0 tx=0).stop 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 msgr2=0x7fd5b4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009230 tx=0x7fd59c004360 comp rx=0 tx=0).stop 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 shutdown_connections 2024-04-24T23:45:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5a00a2b90 0x7fd5a00a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5b4103340 0x7fd5b419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5b4102980 0x7fd5b4198000 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 --2- 172.21.15.107:0/2382541370 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5b4101790 0x7fd5b4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 >> 172.21.15.107:0/2382541370 conn(0x7fd5b40fcd20 msgr2=0x7fd5b4105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 shutdown_connections 2024-04-24T23:45:05.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.534+0000 7fd5bad70640 1 -- 172.21.15.107:0/2382541370 wait complete. 2024-04-24T23:45:05.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:45:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:05 smithi195 ceph-mon[33019]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1165180984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:05 smithi170 ceph-mon[32714]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1165180984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- 172.21.15.107:0/1435436424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190100690 msgr2=0x7fd190100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 --2- 172.21.15.107:0/1435436424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190100690 0x7fd190100ab0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd188009900 tx=0x7fd18802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- 172.21.15.107:0/1435436424 shutdown_connections 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 --2- 172.21.15.107:0/1435436424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190102240 0x7fd190108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 --2- 172.21.15.107:0/1435436424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 0x7fd190101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 --2- 172.21.15.107:0/1435436424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190100690 0x7fd190100ab0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- 172.21.15.107:0/1435436424 >> 172.21.15.107:0/1435436424 conn(0x7fd1900fbc00 msgr2=0x7fd1900fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- 172.21.15.107:0/1435436424 shutdown_connections 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- 172.21.15.107:0/1435436424 wait complete. 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 Processor -- start 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.647+0000 7fd195509640 1 -- start start 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190100690 0x7fd19006a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 0x7fd19019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010c310 con 0x7fd190102240 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010c190 con 0x7fd190101880 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010c490 con 0x7fd190100690 2024-04-24T23:45:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38006/0 (socket says 172.21.15.107:38006) 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 -- 172.21.15.107:0/1769821029 learned_addr learned my addr 172.21.15.107:0/1769821029 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18effd640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190100690 0x7fd19006a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18e7fc640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 0x7fd19019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 -- 172.21.15.107:0/1769821029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190100690 msgr2=0x7fd19006a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190100690 0x7fd19006a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 -- 172.21.15.107:0/1769821029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 msgr2=0x7fd19019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 0x7fd19019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 -- 172.21.15.107:0/1769821029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19019f790 con 0x7fd190102240 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd18f7fe640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd18000cad0 tx=0x7fd18000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000ae20 con 0x7fd190102240 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd180005020 con 0x7fd190102240 2024-04-24T23:45:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19019fa20 con 0x7fd190102240 2024-04-24T23:45:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000f6f0 con 0x7fd190102240 2024-04-24T23:45:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.648+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19019ffc0 con 0x7fd190102240 2024-04-24T23:45:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.649+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd190107a50 con 0x7fd190102240 2024-04-24T23:45:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.650+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd1800051d0 con 0x7fd190102240 2024-04-24T23:45:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.651+0000 7fd177fff640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 0x7fd1780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.651+0000 7fd18effd640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 0x7fd1780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.651+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd1800cf770 con 0x7fd190102240 2024-04-24T23:45:05.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.651+0000 7fd18effd640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 0x7fd1780a50d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd188006010 tx=0x7fd188000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.654+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd180014030 con 0x7fd190102240 2024-04-24T23:45:05.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.828+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd19002d3c0 con 0x7fd190102240 2024-04-24T23:45:05.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.828+0000 7fd177fff640 1 -- 172.21.15.107:0/1769821029 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd18009c380 con 0x7fd190102240 2024-04-24T23:45:05.828 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:45:05.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 msgr2=0x7fd1780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 0x7fd1780a50d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd188006010 tx=0x7fd188000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 msgr2=0x7fd19019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd18000cad0 tx=0x7fd18000cf90 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 shutdown_connections 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.830+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1780a2c10 0x7fd1780a50d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd190102240 0x7fd19019ee80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd190101880 0x7fd19019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 --2- 172.21.15.107:0/1769821029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd190100690 0x7fd19006a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 >> 172.21.15.107:0/1769821029 conn(0x7fd1900fbc00 msgr2=0x7fd190104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 shutdown_connections 2024-04-24T23:45:05.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.831+0000 7fd195509640 1 -- 172.21.15.107:0/1769821029 wait complete. 2024-04-24T23:45:05.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.942+0000 7f3978a2c640 1 -- 172.21.15.107:0/856017211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974101790 msgr2=0x7f3974101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.942+0000 7f3978a2c640 1 --2- 172.21.15.107:0/856017211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974101790 0x7f3974101bb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f396c009ca0 tx=0x7f396c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 -- 172.21.15.107:0/856017211 shutdown_connections 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 --2- 172.21.15.107:0/856017211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974103340 0x7f3974109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 --2- 172.21.15.107:0/856017211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 0x7f3974102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 --2- 172.21.15.107:0/856017211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974101790 0x7f3974101bb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 -- 172.21.15.107:0/856017211 >> 172.21.15.107:0/856017211 conn(0x7f39740fcd20 msgr2=0x7f39740ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 -- 172.21.15.107:0/856017211 shutdown_connections 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 -- 172.21.15.107:0/856017211 wait complete. 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 Processor -- start 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 -- start start 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974101790 0x7f3974197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.943+0000 7f3978a2c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 0x7f3974197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f397410d3e0 con 0x7f3974103340 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f397410d260 con 0x7f3974102980 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f397410d560 con 0x7f3974101790 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38034/0 (socket says 172.21.15.107:38034) 2024-04-24T23:45:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 -- 172.21.15.107:0/319916481 learned_addr learned my addr 172.21.15.107:0/319916481 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3971d9b640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 0x7f3974197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f397259c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974101790 0x7f3974197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 -- 172.21.15.107:0/319916481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974101790 msgr2=0x7f3974197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974101790 0x7f3974197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 -- 172.21.15.107:0/319916481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 msgr2=0x7f3974197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 0x7f3974197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 -- 172.21.15.107:0/319916481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397419f6a0 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3972d9d640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f396400c970 tx=0x7f396400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964007bb0 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397419f990 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964007d50 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964018800 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.944+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397419fee0 con 0x7f3974103340 2024-04-24T23:45:05.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.945+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3974108b30 con 0x7f3974103340 2024-04-24T23:45:05.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.946+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f396400fa50 con 0x7f3974103340 2024-04-24T23:45:05.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.947+0000 7f39637fe640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 0x7f395c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:05.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.947+0000 7f397259c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 0x7f395c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:05.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.947+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39640cf510 con 0x7f3974103340 2024-04-24T23:45:05.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.947+0000 7f397259c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 0x7f395c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f396c006010 tx=0x7f396c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:05.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:05.952+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3964014030 con 0x7f3974103340 2024-04-24T23:45:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.130+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3974062340 con 0x7f3974103340 2024-04-24T23:45:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.130+0000 7f39637fe640 1 -- 172.21.15.107:0/319916481 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f396409c0a0 con 0x7f3974103340 2024-04-24T23:45:06.130 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 msgr2=0x7f395c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 0x7f395c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f396c006010 tx=0x7f396c000f40 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 msgr2=0x7f397419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f396400c970 tx=0x7f396400ce30 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 shutdown_connections 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f395c0a2b90 0x7f395c0a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3974103340 0x7f397419f030 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3974102980 0x7f3974197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 --2- 172.21.15.107:0/319916481 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3974101790 0x7f3974197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 >> 172.21.15.107:0/319916481 conn(0x7f39740fcd20 msgr2=0x7f3974105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 shutdown_connections 2024-04-24T23:45:06.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.133+0000 7f3978a2c640 1 -- 172.21.15.107:0/319916481 wait complete. 2024-04-24T23:45:06.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- 172.21.15.107:0/108891144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58102e50 msgr2=0x7efd58103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/108891144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58102e50 0x7efd58103270 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd48009900 tx=0x7efd4802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- 172.21.15.107:0/108891144 shutdown_connections 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/108891144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd58100280 0x7efd58109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/108891144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd580ff8c0 0x7efd580ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/108891144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58102e50 0x7efd58103270 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- 172.21.15.107:0/108891144 >> 172.21.15.107:0/108891144 conn(0x7efd580fb420 msgr2=0x7efd580fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- 172.21.15.107:0/108891144 shutdown_connections 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- 172.21.15.107:0/108891144 wait complete. 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 Processor -- start 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.242+0000 7efd5ce1b640 1 -- start start 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd580ff8c0 0x7efd58197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd58102e50 0x7efd5819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810d480 con 0x7efd58100280 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810d300 con 0x7efd580ff8c0 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810d600 con 0x7efd58102e50 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38042/0 (socket says 172.21.15.107:38042) 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 -- 172.21.15.107:0/2557009724 learned_addr learned my addr 172.21.15.107:0/2557009724 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5659c640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd580ff8c0 0x7efd58197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd56d9d640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd58102e50 0x7efd5819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 -- 172.21.15.107:0/2557009724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd58102e50 msgr2=0x7efd5819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd58102e50 0x7efd5819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 -- 172.21.15.107:0/2557009724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd580ff8c0 msgr2=0x7efd58197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd580ff8c0 0x7efd58197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 -- 172.21.15.107:0/2557009724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5819f740 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd55d9b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00c890 tx=0x7efd4c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c004220 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd4c0043c0 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5819fa30 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c0049e0 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.243+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5819ffd0 con 0x7efd58100280 2024-04-24T23:45:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.244+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd58108d30 con 0x7efd58100280 2024-04-24T23:45:06.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.245+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7efd4c0079b0 con 0x7efd58100280 2024-04-24T23:45:06.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.246+0000 7efd477fe640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 0x7efd400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.246+0000 7efd5659c640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 0x7efd400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.246+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efd4c0cf390 con 0x7efd58100280 2024-04-24T23:45:06.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.247+0000 7efd5659c640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 0x7efd400a50d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efd48005ab0 tx=0x7efd48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.250+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd4c014030 con 0x7efd58100280 2024-04-24T23:45:06.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3380412511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2382541370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:06 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1769821029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.464+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd58062340 con 0x7efd58100280 2024-04-24T23:45:06.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.464+0000 7efd477fe640 1 -- 172.21.15.107:0/2557009724 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7efd4c09bfc0 con 0x7efd58100280 2024-04-24T23:45:06.464 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:06.464 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:45:06.464 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:45:06.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 msgr2=0x7efd400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 0x7efd400a50d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efd48005ab0 tx=0x7efd48000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 msgr2=0x7efd58198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00c890 tx=0x7efd4c00cd50 comp rx=0 tx=0).stop 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 shutdown_connections 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd400a2c10 0x7efd400a50d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efd58102e50 0x7efd5819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd58100280 0x7efd58198060 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 --2- 172.21.15.107:0/2557009724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd580ff8c0 0x7efd58197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 >> 172.21.15.107:0/2557009724 conn(0x7efd580fb420 msgr2=0x7efd58105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 shutdown_connections 2024-04-24T23:45:06.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.466+0000 7efd5ce1b640 1 -- 172.21.15.107:0/2557009724 wait complete. 2024-04-24T23:45:06.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3380412511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2382541370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:06 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1769821029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 -- 172.21.15.107:0/764187867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 msgr2=0x7fe29c104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/764187867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c104df0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009900 tx=0x7fe28c02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 -- 172.21.15.107:0/764187867 shutdown_connections 2024-04-24T23:45:06.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/764187867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c105330 0x7fe29c1057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/764187867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c104df0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/764187867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c100010 0x7fe29c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 -- 172.21.15.107:0/764187867 >> 172.21.15.107:0/764187867 conn(0x7fe29c0fbc30 msgr2=0x7fe29c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 -- 172.21.15.107:0/764187867 shutdown_connections 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.577+0000 7fe2a36a5640 1 -- 172.21.15.107:0/764187867 wait complete. 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 Processor -- start 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 -- start start 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 0x7fe29c19e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c105330 0x7fe29c1a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c113a10 con 0x7fe29c102980 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c113890 con 0x7fe29c100010 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a36a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c113b90 con 0x7fe29c105330 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57616/0 (socket says 172.21.15.107:57616) 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 -- 172.21.15.107:0/3650622828 learned_addr learned my addr 172.21.15.107:0/3650622828 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a1c42640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c105330 0x7fe29c1a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 -- 172.21.15.107:0/3650622828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c105330 msgr2=0x7fe29c1a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c105330 0x7fe29c1a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 -- 172.21.15.107:0/3650622828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 msgr2=0x7fe29c19e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a1441640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 0x7fe29c19e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 0x7fe29c19e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.578+0000 7fe2a0c40640 1 -- 172.21.15.107:0/3650622828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c1a5d00 con 0x7fe29c102980 2024-04-24T23:45:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2a1441640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 0x7fe29c19e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2a0c40640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009230 tx=0x7fe28c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c03f030 con 0x7fe29c102980 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe28c003ce0 con 0x7fe29c102980 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c035070 con 0x7fe29c102980 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c1a5f90 con 0x7fe29c102980 2024-04-24T23:45:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.579+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c1a63f0 con 0x7fe29c102980 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.580+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c0fd720 con 0x7fe29c102980 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.581+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe28c042030 con 0x7fe29c102980 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.581+0000 7fe2927fc640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 0x7fe2880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.581+0000 7fe2a1441640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 0x7fe2880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.582+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe28c0efc10 con 0x7fe29c102980 2024-04-24T23:45:06.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.582+0000 7fe2a1441640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 0x7fe2880a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe2980058c0 tx=0x7fe298005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3380412511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2382541370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:06 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1769821029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:06.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.586+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe28c0bc7a0 con 0x7fe29c102980 2024-04-24T23:45:06.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.762+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe29c062440 con 0x7fe29c102980 2024-04-24T23:45:06.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.763+0000 7fe2927fc640 1 -- 172.21.15.107:0/3650622828 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7fe28c0bc0a0 con 0x7fe29c102980 2024-04-24T23:45:06.763 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:06.763 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 msgr2=0x7fe2880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 0x7fe2880a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe2980058c0 tx=0x7fe298005d30 comp rx=0 tx=0).stop 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 msgr2=0x7fe29c19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009230 tx=0x7fe28c004430 comp rx=0 tx=0).stop 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 shutdown_connections 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fe2880a2b90 0x7fe2880a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe29c105330 0x7fe29c1a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe29c102980 0x7fe29c19e620 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 --2- 172.21.15.107:0/3650622828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29c100010 0x7fe29c19e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 >> 172.21.15.107:0/3650622828 conn(0x7fe29c0fbc30 msgr2=0x7fe29c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 shutdown_connections 2024-04-24T23:45:06.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.765+0000 7fe2a36a5640 1 -- 172.21.15.107:0/3650622828 wait complete. 2024-04-24T23:45:06.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:45:06.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- 172.21.15.107:0/1149628459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c101790 msgr2=0x7f6e4c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 --2- 172.21.15.107:0/1149628459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c101790 0x7f6e4c101bb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009900 tx=0x7f6e4802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- 172.21.15.107:0/1149628459 shutdown_connections 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 --2- 172.21.15.107:0/1149628459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 0x7f6e4c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 --2- 172.21.15.107:0/1149628459 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c102980 0x7f6e4c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 --2- 172.21.15.107:0/1149628459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c101790 0x7f6e4c101bb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- 172.21.15.107:0/1149628459 >> 172.21.15.107:0/1149628459 conn(0x7f6e4c0fcd20 msgr2=0x7f6e4c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:06.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- 172.21.15.107:0/1149628459 shutdown_connections 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- 172.21.15.107:0/1149628459 wait complete. 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 Processor -- start 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.878+0000 7f6e53c58640 1 -- start start 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c101790 0x7f6e4c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 0x7f6e4c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c10d400 con 0x7f6e4c102980 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c10d280 con 0x7f6e4c103340 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c10d580 con 0x7f6e4c101790 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57620/0 (socket says 172.21.15.107:57620) 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 -- 172.21.15.107:0/3556370151 learned_addr learned my addr 172.21.15.107:0/3556370151 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:06.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e519f4640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c101790 0x7f6e4c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e521f5640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 0x7f6e4c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 -- 172.21.15.107:0/3556370151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c101790 msgr2=0x7f6e4c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c101790 0x7f6e4c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 -- 172.21.15.107:0/3556370151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 msgr2=0x7f6e4c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 0x7f6e4c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 -- 172.21.15.107:0/3556370151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c074900 con 0x7f6e4c102980 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e511f3640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d770 tx=0x7f6e3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c017070 con 0x7f6e4c102980 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4c1a1bf0 con 0x7f6e4c102980 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e3c005500 con 0x7f6e4c102980 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c004e00 con 0x7f6e4c102980 2024-04-24T23:45:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.879+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4c1a2140 con 0x7f6e4c102980 2024-04-24T23:45:06.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.880+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e4c108bd0 con 0x7f6e4c102980 2024-04-24T23:45:06.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.881+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6e3c005000 con 0x7f6e4c102980 2024-04-24T23:45:06.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.882+0000 7f6e42ffd640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 0x7f6e380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:06.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.882+0000 7f6e519f4640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 0x7f6e380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:06.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.882+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6e3c0cee80 con 0x7f6e4c102980 2024-04-24T23:45:06.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.882+0000 7f6e519f4640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 0x7f6e380a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005ab0 tx=0x7f6e48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:06.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:06.886+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e3c09ba10 con 0x7f6e4c102980 2024-04-24T23:45:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.060+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e4c062340 con 0x7f6e4c102980 2024-04-24T23:45:07.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.061+0000 7f6e42ffd640 1 -- 172.21.15.107:0/3556370151 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f6e3c09b310 con 0x7f6e4c102980 2024-04-24T23:45:07.061 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:45:07.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 msgr2=0x7f6e380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 0x7f6e380a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005ab0 tx=0x7f6e48000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 msgr2=0x7f6e4c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d770 tx=0x7f6e3c00dc30 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 shutdown_connections 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6e380a2b90 0x7f6e380a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e4c103340 0x7f6e4c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e4c102980 0x7f6e4c073b30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 --2- 172.21.15.107:0/3556370151 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6e4c101790 0x7f6e4c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 >> 172.21.15.107:0/3556370151 conn(0x7f6e4c0fcd20 msgr2=0x7f6e4c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 shutdown_connections 2024-04-24T23:45:07.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.063+0000 7f6e53c58640 1 -- 172.21.15.107:0/3556370151 wait complete. 2024-04-24T23:45:07.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4192896760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38107460 msgr2=0x7f7b381098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4192896760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38107460 0x7f7b381098b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009ca0 tx=0x7f7b3002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4192896760 shutdown_connections 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4192896760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38107460 0x7f7b381098b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4192896760 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 0x7f7b38106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4192896760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38102200 0x7f7b381045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4192896760 >> 172.21.15.107:0/4192896760 conn(0x7f7b380fbbf0 msgr2=0x7f7b380fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4192896760 shutdown_connections 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.176+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4192896760 wait complete. 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 Processor -- start 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 -- start start 2024-04-24T23:45:07.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 0x7f7b38198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38107460 0x7f7b3819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810d470 con 0x7f7b38102200 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810d2f0 con 0x7f7b38107460 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810d5f0 con 0x7f7b38104b30 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57634/0 (socket says 172.21.15.107:57634) 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 -- 172.21.15.107:0/4254323284 learned_addr learned my addr 172.21.15.107:0/4254323284 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b37fff640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 0x7f7b38198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 -- 172.21.15.107:0/4254323284 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 msgr2=0x7f7b38198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 0x7f7b38198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3d165640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38107460 0x7f7b3819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 -- 172.21.15.107:0/4254323284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38107460 msgr2=0x7f7b3819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38107460 0x7f7b3819f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 -- 172.21.15.107:0/4254323284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3819f690 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3c964640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c920 tx=0x7f7b2c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c007c70 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b2c007e10 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.177+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3819f980 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.178+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c018890 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.178+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3819ff50 con 0x7f7b38102200 2024-04-24T23:45:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.179+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b380fd6e0 con 0x7f7b38102200 2024-04-24T23:45:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.180+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7b2c018a70 con 0x7f7b38102200 2024-04-24T23:45:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.180+0000 7f7b35ffb640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 0x7f7b280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.180+0000 7f7b37fff640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 0x7f7b280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.181+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7b2c0cf610 con 0x7f7b38102200 2024-04-24T23:45:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.181+0000 7f7b37fff640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 0x7f7b280a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7b38198f80 tx=0x7f7b24008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.185+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b2c014030 con 0x7f7b38102200 2024-04-24T23:45:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.361+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7b38062340 con 0x7f7b38102200 2024-04-24T23:45:07.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.362+0000 7f7b35ffb640 1 -- 172.21.15.107:0/4254323284 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f7b2c00de20 con 0x7f7b38102200 2024-04-24T23:45:07.362 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 msgr2=0x7f7b280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 0x7f7b280a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7b38198f80 tx=0x7f7b24008040 comp rx=0 tx=0).stop 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 msgr2=0x7f7b38197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c920 tx=0x7f7b2c00cde0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 shutdown_connections 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b280a2b90 0x7f7b280a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b38107460 0x7f7b3819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b38104b30 0x7f7b38198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 --2- 172.21.15.107:0/4254323284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38102200 0x7f7b38197ac0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 >> 172.21.15.107:0/4254323284 conn(0x7f7b380fbbf0 msgr2=0x7f7b38102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 shutdown_connections 2024-04-24T23:45:07.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.364+0000 7f7b3ebc8640 1 -- 172.21.15.107:0/4254323284 wait complete. 2024-04-24T23:45:07.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:07 smithi107 ceph-mon[29118]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:07.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/319916481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2557009724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3650622828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3556370151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- 172.21.15.107:0/2996590845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150102200 msgr2=0x7ff1501045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 --2- 172.21.15.107:0/2996590845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150102200 0x7ff1501045f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff140009900 tx=0x7ff14002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- 172.21.15.107:0/2996590845 shutdown_connections 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 --2- 172.21.15.107:0/2996590845 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150107460 0x7ff1501098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 --2- 172.21.15.107:0/2996590845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150104b30 0x7ff150106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 --2- 172.21.15.107:0/2996590845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150102200 0x7ff1501045f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- 172.21.15.107:0/2996590845 >> 172.21.15.107:0/2996590845 conn(0x7ff1500fbbf0 msgr2=0x7ff1500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- 172.21.15.107:0/2996590845 shutdown_connections 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- 172.21.15.107:0/2996590845 wait complete. 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 Processor -- start 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.476+0000 7ff156784640 1 -- start start 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 0x7ff150197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150104b30 0x7ff1501980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d470 con 0x7ff150107460 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d2f0 con 0x7ff150102200 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff156784640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d5f0 con 0x7ff150104b30 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57660/0 (socket says 172.21.15.107:57660) 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 -- 172.21.15.107:0/1408955237 learned_addr learned my addr 172.21.15.107:0/1408955237 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff14ffff640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 0x7ff150197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff14f7fe640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150104b30 0x7ff1501980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 -- 172.21.15.107:0/1408955237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150104b30 msgr2=0x7ff1501980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150104b30 0x7ff1501980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 -- 172.21.15.107:0/1408955237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 msgr2=0x7ff150197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 0x7ff150197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 -- 172.21.15.107:0/1408955237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15019f790 con 0x7ff150107460 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff14ffff640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 0x7ff150197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff154d21640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff14800b970 tx=0x7ff14800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148019070 con 0x7ff150107460 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.477+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff14800cb40 con 0x7ff150107460 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.478+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15019fa80 con 0x7ff150107460 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.478+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148007650 con 0x7ff150107460 2024-04-24T23:45:07.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.478+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15019ffd0 con 0x7ff150107460 2024-04-24T23:45:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.478+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1500fd6e0 con 0x7ff150107460 2024-04-24T23:45:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.479+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff14801c070 con 0x7ff150107460 2024-04-24T23:45:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.480+0000 7ff14d7fa640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 0x7ff1380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.480+0000 7ff14ffff640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 0x7ff1380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.480+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff1480cea20 con 0x7ff150107460 2024-04-24T23:45:07.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.481+0000 7ff14ffff640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 0x7ff1380a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff140000c00 tx=0x7ff1400315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.485+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff14809b5b0 con 0x7ff150107460 2024-04-24T23:45:07.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:07 smithi170 ceph-mon[32714]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:07.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/319916481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2557009724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3650622828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3556370151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:07 smithi195 ceph-mon[33019]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/319916481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2557009724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3650622828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3556370151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:07.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.658+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff150062340 con 0x7ff150107460 2024-04-24T23:45:07.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.658+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1408955237 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ff14809aeb0 con 0x7ff150107460 2024-04-24T23:45:07.658 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 msgr2=0x7ff1380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 0x7ff1380a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff140000c00 tx=0x7ff1400315d0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 msgr2=0x7ff15019f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff14800b970 tx=0x7ff14800be30 comp rx=0 tx=0).stop 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 shutdown_connections 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1380a2b90 0x7ff1380a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff150107460 0x7ff15019f120 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff150104b30 0x7ff1501980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 --2- 172.21.15.107:0/1408955237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff150102200 0x7ff150197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 >> 172.21.15.107:0/1408955237 conn(0x7ff1500fbbf0 msgr2=0x7ff150102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 shutdown_connections 2024-04-24T23:45:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.660+0000 7ff156784640 1 -- 172.21.15.107:0/1408955237 wait complete. 2024-04-24T23:45:07.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:45:07.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 -- 172.21.15.107:0/665032452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0101880 msgr2=0x7fb6e0101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 --2- 172.21.15.107:0/665032452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0101880 0x7fb6e0101d00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009900 tx=0x7fb6c802a3f0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 -- 172.21.15.107:0/665032452 shutdown_connections 2024-04-24T23:45:07.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 --2- 172.21.15.107:0/665032452 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0102240 0x7fb6e0108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 --2- 172.21.15.107:0/665032452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0101880 0x7fb6e0101d00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 --2- 172.21.15.107:0/665032452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0100690 0x7fb6e0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 -- 172.21.15.107:0/665032452 >> 172.21.15.107:0/665032452 conn(0x7fb6e00fbc00 msgr2=0x7fb6e00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 -- 172.21.15.107:0/665032452 shutdown_connections 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.772+0000 7fb6e549a640 1 -- 172.21.15.107:0/665032452 wait complete. 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 Processor -- start 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 -- start start 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 0x7fb6e0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0101880 0x7fb6e0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e010d430 con 0x7fb6e0102240 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e010d2b0 con 0x7fb6e0101880 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6e549a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e010d5b0 con 0x7fb6e0100690 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57666/0 (socket says 172.21.15.107:57666) 2024-04-24T23:45:07.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 -- 172.21.15.107:0/876189765 learned_addr learned my addr 172.21.15.107:0/876189765 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6d7fff640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0101880 0x7fb6e0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6deffd640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 0x7fb6e0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 -- 172.21.15.107:0/876189765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 msgr2=0x7fb6e0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 0x7fb6e0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 -- 172.21.15.107:0/876189765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0101880 msgr2=0x7fb6e0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0101880 0x7fb6e0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 -- 172.21.15.107:0/876189765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e019f6d0 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6deffd640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 0x7fb6e0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6df7fe640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800eaa0 tx=0x7fb6d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d800cdb0 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.773+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6d8005020 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.774+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e019f9c0 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.774+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80116b0 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.774+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e019fee0 con 0x7fb6e0102240 2024-04-24T23:45:07.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.774+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e0107a30 con 0x7fb6e0102240 2024-04-24T23:45:07.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.776+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb6d8005560 con 0x7fb6e0102240 2024-04-24T23:45:07.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.776+0000 7fb6dcff9640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 0x7fb6cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:07.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.776+0000 7fb6deffd640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 0x7fb6cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:07.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.776+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb6d80cfa90 con 0x7fb6e0102240 2024-04-24T23:45:07.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.777+0000 7fb6deffd640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 0x7fb6cc0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6e01016e0 tx=0x7fb6d0005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:07.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.780+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb6d8016030 con 0x7fb6e0102240 2024-04-24T23:45:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.954+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb6e002d170 con 0x7fb6e0102240 2024-04-24T23:45:07.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.954+0000 7fb6dcff9640 1 -- 172.21.15.107:0/876189765 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb6d809c620 con 0x7fb6e0102240 2024-04-24T23:45:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 msgr2=0x7fb6cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 0x7fb6cc0a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6e01016e0 tx=0x7fb6d0005e70 comp rx=0 tx=0).stop 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 msgr2=0x7fb6e019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800eaa0 tx=0x7fb6d800ef60 comp rx=0 tx=0).stop 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 shutdown_connections 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb6cc0a2b90 0x7fb6cc0a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6e0102240 0x7fb6e019f060 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6e0101880 0x7fb6e0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 --2- 172.21.15.107:0/876189765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb6e0100690 0x7fb6e0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 >> 172.21.15.107:0/876189765 conn(0x7fb6e00fbc00 msgr2=0x7fb6e0104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 shutdown_connections 2024-04-24T23:45:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:07.956+0000 7fb6e549a640 1 -- 172.21.15.107:0/876189765 wait complete. 2024-04-24T23:45:07.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:45:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 -- 172.21.15.107:0/2381679756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b8101890 msgr2=0x7fb9b8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 --2- 172.21.15.107:0/2381679756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b8101890 0x7fb9b8101d10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009900 tx=0x7fb9a802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 -- 172.21.15.107:0/2381679756 shutdown_connections 2024-04-24T23:45:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 --2- 172.21.15.107:0/2381679756 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9b8102250 0x7fb9b8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 --2- 172.21.15.107:0/2381679756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b8101890 0x7fb9b8101d10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 --2- 172.21.15.107:0/2381679756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b81006a0 0x7fb9b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 -- 172.21.15.107:0/2381679756 >> 172.21.15.107:0/2381679756 conn(0x7fb9b80fbc30 msgr2=0x7fb9b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 -- 172.21.15.107:0/2381679756 shutdown_connections 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.065+0000 7fb9c0288640 1 -- 172.21.15.107:0/2381679756 wait complete. 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 Processor -- start 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- start start 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9b8101890 0x7fb9b8102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8102250 0x7fb9b81032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b810c300 con 0x7fb9b81006a0 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b810c180 con 0x7fb9b8102250 2024-04-24T23:45:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b810c480 con 0x7fb9b8101890 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57682/0 (socket says 172.21.15.107:57682) 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 -- 172.21.15.107:0/1453710900 learned_addr learned my addr 172.21.15.107:0/1453710900 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 -- 172.21.15.107:0/1453710900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9b8101890 msgr2=0x7fb9b8102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9b8101890 0x7fb9b8102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 -- 172.21.15.107:0/1453710900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8102250 msgr2=0x7fb9b81032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8102250 0x7fb9b81032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 -- 172.21.15.107:0/1453710900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b8103b30 con 0x7fb9b81006a0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9be024640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400d950 tx=0x7fb9b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4017020 con 0x7fb9b81006a0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b4009c30 con 0x7fb9b81006a0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b81a1be0 con 0x7fb9b81006a0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b400f740 con 0x7fb9b81006a0 2024-04-24T23:45:08.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.066+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b81a2130 con 0x7fb9b81006a0 2024-04-24T23:45:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.068+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b81a2360 con 0x7fb9b81006a0 2024-04-24T23:45:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.069+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb9b401a020 con 0x7fb9b81006a0 2024-04-24T23:45:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.069+0000 7fb9af7fe640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 0x7fb9a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.069+0000 7fb9bd823640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 0x7fb9a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.069+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb9b40cef80 con 0x7fb9b81006a0 2024-04-24T23:45:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.070+0000 7fb9bd823640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 0x7fb9a40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80096c0 tx=0x7fb9a8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.074+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb9b4021070 con 0x7fb9b81006a0 2024-04-24T23:45:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.286+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9b81a2c00 con 0x7fb9b81006a0 2024-04-24T23:45:08.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.287+0000 7fb9af7fe640 1 -- 172.21.15.107:0/1453710900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11956 (secure 0 0 0) 0x7fb9b4009de0 con 0x7fb9b81006a0 2024-04-24T23:45:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:08.287 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:43:19.093489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24464,24506,24463,24493,24451]},"id":1}]} 2024-04-24T23:45:08.287 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-04-24T23:45:08.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.288+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 msgr2=0x7fb9a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.288+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 0x7fb9a40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80096c0 tx=0x7fb9a8031400 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.288+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 msgr2=0x7fb9b8104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.288+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400d950 tx=0x7fb9b400de10 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 shutdown_connections 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9a40a2b90 0x7fb9a40a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8102250 0x7fb9b81032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb9b8101890 0x7fb9b8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 --2- 172.21.15.107:0/1453710900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9b81006a0 0x7fb9b8104720 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 >> 172.21.15.107:0/1453710900 conn(0x7fb9b80fbc30 msgr2=0x7fb9b8069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 shutdown_connections 2024-04-24T23:45:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.289+0000 7fb9c0288640 1 -- 172.21.15.107:0/1453710900 wait complete. 2024-04-24T23:45:08.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4254323284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1408955237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/876189765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 -- 172.21.15.107:0/3765028258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 msgr2=0x7fd6341046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3765028258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd6341046b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd624009ca0 tx=0x7fd62402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 -- 172.21.15.107:0/3765028258 shutdown_connections 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3765028258 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 0x7fd634109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3765028258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 0x7fd634106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.398+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3765028258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd6341046b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- 172.21.15.107:0/3765028258 >> 172.21.15.107:0/3765028258 conn(0x7fd6340fbbf0 msgr2=0x7fd6340fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- 172.21.15.107:0/3765028258 shutdown_connections 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- 172.21.15.107:0/3765028258 wait complete. 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 Processor -- start 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- start start 2024-04-24T23:45:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 0x7fd634100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 0x7fd6341007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d480 con 0x7fd6341022c0 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d300 con 0x7fd634104bf0 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.399+0000 7fd638bdc640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d600 con 0x7fd634107520 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57700/0 (socket says 172.21.15.107:57700) 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 -- 172.21.15.107:0/3110166309 learned_addr learned my addr 172.21.15.107:0/3110166309 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd631d9b640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 0x7fd634100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd632d9d640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 0x7fd6341007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 -- 172.21.15.107:0/3110166309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 msgr2=0x7fd6341007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 0x7fd6341007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 -- 172.21.15.107:0/3110166309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 msgr2=0x7fd634100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 0x7fd634100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 -- 172.21.15.107:0/3110166309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd634101040 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd63259c640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd624031a40 tx=0x7fd624031ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62402faa0 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6341a1c40 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd62402fc40 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62403a790 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.400+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6341a2190 con 0x7fd6341022c0 2024-04-24T23:45:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.401+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6340fd6e0 con 0x7fd6341022c0 2024-04-24T23:45:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.402+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd624030b30 con 0x7fd6341022c0 2024-04-24T23:45:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.403+0000 7fd6237fe640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 0x7fd61c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.403+0000 7fd631d9b640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 0x7fd61c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.403+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6240f0b40 con 0x7fd6341022c0 2024-04-24T23:45:08.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.403+0000 7fd631d9b640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 0x7fd61c0a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd6341019a0 tx=0x7fd62800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.407+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6240bd6d0 con 0x7fd6341022c0 2024-04-24T23:45:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4254323284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1408955237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/876189765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.583+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd634062370 con 0x7fd6341022c0 2024-04-24T23:45:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.585+0000 7fd6237fe640 1 -- 172.21.15.107:0/3110166309 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7fd6240bcfd0 con 0x7fd6341022c0 2024-04-24T23:45:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:08.585 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:08.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4254323284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1408955237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/876189765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 msgr2=0x7fd61c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 0x7fd61c0a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd6341019a0 tx=0x7fd62800b390 comp rx=0 tx=0).stop 2024-04-24T23:45:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 msgr2=0x7fd634101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd624031a40 tx=0x7fd624031ec0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 shutdown_connections 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd61c0a2b90 0x7fd61c0a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd634107520 0x7fd6341007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd634104bf0 0x7fd634100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 --2- 172.21.15.107:0/3110166309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6341022c0 0x7fd634101c30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 >> 172.21.15.107:0/3110166309 conn(0x7fd6340fbbf0 msgr2=0x7fd634102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 shutdown_connections 2024-04-24T23:45:08.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.587+0000 7fd638bdc640 1 -- 172.21.15.107:0/3110166309 wait complete. 2024-04-24T23:45:08.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T23:42:50.405146+0000', 'modified': '2024-04-24T23:43:19.093489+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24493, 'mds_2': 24464, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6839/527084969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 527084969}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 527084969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6841/1612113576', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6840', 'nonce': 1612113576}, {'type': 'v1', 'addr': '172.21.15.107:6841', 'nonce': 1612113576}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24406': {'gid': 24406, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.195:6833/1442081594', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 1442081594}, {'type': 'v1', 'addr': '172.21.15.195:6833', 'nonce': 1442081594}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6833/3444431424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3444431424}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 3444431424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/3179503985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 3179503985}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3179503985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.195:6835/1997811297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6834', 'nonce': 1997811297}, {'type': 'v1', 'addr': '172.21.15.195:6835', 'nonce': 1997811297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.195:6837/1315131786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6836', 'nonce': 1315131786}, {'type': 'v1', 'addr': '172.21.15.195:6837', 'nonce': 1315131786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.170:6837/293399389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 293399389}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 293399389}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.195:6839/4197811615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6838', 'nonce': 4197811615}, {'type': 'v1', 'addr': '172.21.15.195:6839', 'nonce': 4197811615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6839/2021576737', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 2021576737}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 2021576737}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24464, 24506, 24463, 24493, 24451]} 2024-04-24T23:45:08.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-24T23:45:08.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-24T23:45:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 -- 172.21.15.107:0/2160396646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310102200 msgr2=0x7f53101045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 --2- 172.21.15.107:0/2160396646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310102200 0x7f53101045f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f530002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 -- 172.21.15.107:0/2160396646 shutdown_connections 2024-04-24T23:45:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 --2- 172.21.15.107:0/2160396646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310107460 0x7f53101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 --2- 172.21.15.107:0/2160396646 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310104b30 0x7f5310106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 --2- 172.21.15.107:0/2160396646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310102200 0x7f53101045f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 -- 172.21.15.107:0/2160396646 >> 172.21.15.107:0/2160396646 conn(0x7f53100fbc30 msgr2=0x7f53100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 -- 172.21.15.107:0/2160396646 shutdown_connections 2024-04-24T23:45:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.704+0000 7f5314f29640 1 -- 172.21.15.107:0/2160396646 wait complete. 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 Processor -- start 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 -- start start 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310102200 0x7f5310197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310104b30 0x7f5310197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d400 con 0x7f5310107460 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d280 con 0x7f5310104b30 2024-04-24T23:45:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5314f29640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d580 con 0x7f5310102200 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f530ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f530ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57710/0 (socket says 172.21.15.107:57710) 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f530ed9d640 1 -- 172.21.15.107:0/3227846549 learned_addr learned my addr 172.21.15.107:0/3227846549 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f5307fff640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310104b30 0x7f5310197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.705+0000 7f530e59c640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310102200 0x7f5310197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 -- 172.21.15.107:0/3227846549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310102200 msgr2=0x7f5310197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310102200 0x7f5310197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 -- 172.21.15.107:0/3227846549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310104b30 msgr2=0x7f5310197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310104b30 0x7f5310197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 -- 172.21.15.107:0/3227846549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531019f6a0 con 0x7f5310107460 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f530ed9d640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f530800cce0 tx=0x7f530800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f53077fe640 1 -- 172.21.15.107:0/3227846549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530800f450 con 0x7f5310107460 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f53077fe640 1 -- 172.21.15.107:0/3227846549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530800fad0 con 0x7f5310107460 2024-04-24T23:45:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531019f990 con 0x7f5310107460 2024-04-24T23:45:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f53077fe640 1 -- 172.21.15.107:0/3227846549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308018c10 con 0x7f5310107460 2024-04-24T23:45:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.706+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f531019fe90 con 0x7f5310107460 2024-04-24T23:45:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.708+0000 7f53077fe640 1 -- 172.21.15.107:0/3227846549 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f530800f5f0 con 0x7f5310107460 2024-04-24T23:45:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.709+0000 7f53077fe640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 0x7f52fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.709+0000 7f530e59c640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 0x7f52fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.709+0000 7f53077fe640 1 -- 172.21.15.107:0/3227846549 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f53080cf9e0 con 0x7f5310107460 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.709+0000 7f530e59c640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 0x7f52fc0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53000314e0 tx=0x7f5300000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.709+0000 7f5317c38740 1 Processor -- start 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 -- start start 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26bb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26bbe30 0x559ee26c0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ee261ef00 con 0x559ee26c0fa0 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559ee25c7750 con 0x559ee26bb440 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5317c38740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559ee261aec0 con 0x559ee26bbe30 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f530ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f530ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57714/0 (socket says 172.21.15.107:57714) 2024-04-24T23:45:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f530ed9d640 1 -- 172.21.15.107:0/856122102 learned_addr learned my addr 172.21.15.107:0/856122102 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f5307fff640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26bbe30 0x559ee26c0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f530e59c640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.710+0000 7f530ed9d640 1 -- 172.21.15.107:0/856122102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26bbe30 msgr2=0x559ee26c0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530ed9d640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26bbe30 0x559ee26c0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530ed9d640 1 -- 172.21.15.107:0/856122102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 msgr2=0x559ee26bb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530ed9d640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26bb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530ed9d640 1 -- 172.21.15.107:0/856122102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ee23332d0 con 0x559ee26c0fa0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530e59c640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26bb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f530ed9d640 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f530809bb70 tx=0x7f5308097b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce885f377c40ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f5304ff9640 1 -- 172.21.15.107:0/856122102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530809be70 con 0x559ee26c0fa0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f5304ff9640 1 -- 172.21.15.107:0/856122102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53080a0220 con 0x559ee26c0fa0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.711+0000 7f5304ff9640 1 -- 172.21.15.107:0/856122102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080a4650 con 0x559ee26c0fa0 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 -- 172.21.15.107:0/856122102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 msgr2=0x559ee26c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f530809bb70 tx=0x7f5308097b10 comp rx=0 tx=0).stop 2024-04-24T23:45:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 -- 172.21.15.107:0/856122102 shutdown_connections 2024-04-24T23:45:08.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26c0fa0 0x559ee26c3430 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26bbe30 0x559ee26c0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 --2- 172.21.15.107:0/856122102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26bb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 -- 172.21.15.107:0/856122102 >> 172.21.15.107:0/856122102 conn(0x559ee26b6ab0 msgr2=0x559ee26b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 -- 172.21.15.107:0/856122102 shutdown_connections 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.712+0000 7f5317c38740 1 -- 172.21.15.107:0/856122102 wait complete. 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 -- start start 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 Processor -- start 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ee26c3f50 con 0x559ee26bbe30 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559ee25dacf0 con 0x559ee26bb440 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559ee26c40d0 con 0x559ee26c0fa0 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5317c38740 10 client.0 _ll_register_callbacks cb 0x559ee25da6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57720/0 (socket says 172.21.15.107:57720) 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5307fff640 1 -- 172.21.15.107:0/4237921397 learned_addr learned my addr 172.21.15.107:0/4237921397 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f530e59c640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f530ed9d640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5307fff640 1 -- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 msgr2=0x559ee26cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.713+0000 7f5307fff640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5307fff640 1 -- 172.21.15.107:0/4237921397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 msgr2=0x559ee26c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5307fff640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5307fff640 1 -- 172.21.15.107:0/4237921397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ee23fd6b0 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f530ed9d640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5307fff640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005fd0 tx=0x7f52f80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74adc558a6391502 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004020 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f52f80041c0 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53080a0220 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f52f8024050 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5317c38740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ee26cff20 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53080a0220 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.714+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f52f8004c10 con 0x559ee26bbe30 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 10 client.15450 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 20 client.15450 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee26d2630 0x559ee26d3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 4 client.15450 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ee262c440 con 0x559ee26d2630 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f530e59c640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee26d2630 0x559ee26d3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f52f5ffb640 10 client.15450 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.715+0000 7f530e59c640 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee26d2630 0x559ee26d3aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e42f277916d8102c server_cookie=b9605c63e524ac3f in_seq=0 out_seq=0 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f52f5ffb640 1 -- 172.21.15.107:0/4237921397 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ee262c440 con 0x559ee26d2630 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f52f5ffb640 10 client.15450 handle_command_reply: tid=0 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 client.15450 shutdown 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 msgr2=0x559ee26c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005fd0 tx=0x7f52f80118d0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 shutdown_connections 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee26d2630 0x559ee26d3aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c5e90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 --2- 172.21.15.107:0/4237921397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 >> 172.21.15.107:0/4237921397 conn(0x559ee26b6ab0 msgr2=0x559ee26b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f52f5ffb640 0 client.15450 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.716+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 shutdown_connections 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.717+0000 7f5317c38740 1 -- 172.21.15.107:0/4237921397 wait complete. 2024-04-24T23:45:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.717+0000 7f5317c38740 20 client.15450 trim_cache size 0 max 16384 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 Processor -- start 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 -- start start 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26b9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d1de0 con 0x559ee26bbe30 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d24b0 con 0x559ee26bb440 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5317c38740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d2b80 con 0x559ee26c0fa0 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57722/0 (socket says 172.21.15.107:57722) 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f5307fff640 1 -- 172.21.15.107:0/2553725795 learned_addr learned my addr 172.21.15.107:0/2553725795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.747+0000 7f530ed9d640 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f530e59c640 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 -- 172.21.15.107:0/2553725795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 msgr2=0x559ee26b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 -- 172.21.15.107:0/2553725795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 msgr2=0x559ee26b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 -- 172.21.15.107:0/2553725795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ee26b68e0 con 0x559ee26bbe30 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f5307fff640 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f52f8004f80 tx=0x7f52f8005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7fefd0f2b6aabf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f52f7fff640 1 -- 172.21.15.107:0/2553725795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8010a40 con 0x559ee26bbe30 2024-04-24T23:45:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f52f7fff640 1 -- 172.21.15.107:0/2553725795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f80054e0 con 0x559ee26bbe30 2024-04-24T23:45:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.748+0000 7f52f7fff640 1 -- 172.21.15.107:0/2553725795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004c10 con 0x559ee26bbe30 2024-04-24T23:45:08.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- 172.21.15.107:0/2553725795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 msgr2=0x559ee26cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f52f8004f80 tx=0x7f52f8005b90 comp rx=0 tx=0).stop 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- 172.21.15.107:0/2553725795 shutdown_connections 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26cce90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 --2- 172.21.15.107:0/2553725795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- 172.21.15.107:0/2553725795 >> 172.21.15.107:0/2553725795 conn(0x559ee26b6ab0 msgr2=0x559ee26bc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- 172.21.15.107:0/2553725795 shutdown_connections 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- 172.21.15.107:0/2553725795 wait complete. 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.749+0000 7f5317c38740 1 -- start start 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 Processor -- start 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d3920 con 0x559ee26bbe30 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d3250 con 0x559ee26bb440 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559ee26d3ff0 con 0x559ee26c0fa0 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5317c38740 10 client.0 _ll_register_callbacks cb 0x7f52f800bad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57736/0 (socket says 172.21.15.107:57736) 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 -- 172.21.15.107:0/1754674172 learned_addr learned my addr 172.21.15.107:0/1754674172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f530e59c640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f530ed9d640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 -- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 msgr2=0x559ee26cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 -- 172.21.15.107:0/1754674172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 msgr2=0x559ee26c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 -- 172.21.15.107:0/1754674172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ee26cbd30 con 0x559ee26bbe30 2024-04-24T23:45:08.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f5307fff640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f52f800bd20 tx=0x7f52f80052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2b450054d537b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.750+0000 7f52f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8010c50 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f52f8010df0 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52f80054e0 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f52f8004910 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 10 client.15456 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ee26cd030 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f80054e0 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f52f8037070 con 0x559ee26bbe30 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 10 client.15456 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 10 client.15456 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 20 client.15456 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee273cbf0 0x559ee273b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 4 client.15456 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.751+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "status"}) -- 0x559ee2626790 con 0x559ee273cbf0 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f530e59c640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee273cbf0 0x559ee273b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f52f5ffb640 10 client.15456 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f530e59c640 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee273cbf0 0x559ee273b900 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5b58c940d6dba12 server_cookie=fb2e7b8764a9df74 in_seq=0 out_seq=0 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f52f5ffb640 1 -- 172.21.15.107:0/1754674172 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x559ee2626790 con 0x559ee273cbf0 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f52f5ffb640 10 client.15456 handle_command_reply: tid=0 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 client.15456 shutdown 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 msgr2=0x559ee26c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f52f800bd20 tx=0x7f52f80052b0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 shutdown_connections 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x559ee273cbf0 0x559ee273b900 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559ee26c0fa0 0x559ee26cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ee26bbe30 0x559ee26c8720 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f52f5ffb640 0 client.15456 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 --2- 172.21.15.107:0/1754674172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559ee26bb440 0x559ee26c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 >> 172.21.15.107:0/1754674172 conn(0x559ee26b6ab0 msgr2=0x559ee26b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 shutdown_connections 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.752+0000 7f5317c38740 1 -- 172.21.15.107:0/1754674172 wait complete. 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5317c38740 20 client.15456 trim_cache size 0 max 16384 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 4, 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24451, 2024-04-24T23:45:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 110.662186449, 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 161.701990532 2024-04-24T23:45:08.760 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 msgr2=0x7f52fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 0x7f52fc0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53000314e0 tx=0x7f5300000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 msgr2=0x7f531019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f530800cce0 tx=0x7f530800a620 comp rx=0 tx=0).stop 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 shutdown_connections 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f52fc0a2b90 0x7f52fc0a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5310107460 0x7f531019f030 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5310104b30 0x7f5310197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 --2- 172.21.15.107:0/3227846549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5310102200 0x7f5310197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 >> 172.21.15.107:0/3227846549 conn(0x7f53100fbc30 msgr2=0x7f5310102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 shutdown_connections 2024-04-24T23:45:08.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.753+0000 7f5314f29640 1 -- 172.21.15.107:0/3227846549 wait complete. 2024-04-24T23:45:08.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T23:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3286046158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc104b30 msgr2=0x7fa7bc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3286046158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc104b30 0x7fa7bc106f20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009900 tx=0x7fa7a402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3286046158 shutdown_connections 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3286046158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc107460 0x7fa7bc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3286046158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc104b30 0x7fa7bc106f20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3286046158 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc102200 0x7fa7bc1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3286046158 >> 172.21.15.107:0/3286046158 conn(0x7fa7bc0fbbf0 msgr2=0x7fa7bc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3286046158 shutdown_connections 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3286046158 wait complete. 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 Processor -- start 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.864+0000 7fa7c25a9640 1 -- start start 2024-04-24T23:45:08.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc102200 0x7fa7bc101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 0x7fa7bc1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10d410 con 0x7fa7bc107460 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10d290 con 0x7fa7bc102200 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10d590 con 0x7fa7bc104b30 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57760/0 (socket says 172.21.15.107:57760) 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 -- 172.21.15.107:0/3643259913 learned_addr learned my addr 172.21.15.107:0/3643259913 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc102200 0x7fa7bc101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 0x7fa7bc1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 -- 172.21.15.107:0/3643259913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 msgr2=0x7fa7bc1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 0x7fa7bc1001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 -- 172.21.15.107:0/3643259913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc102200 msgr2=0x7fa7bc101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc102200 0x7fa7bc101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 -- 172.21.15.107:0/3643259913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc100f80 con 0x7fa7bc107460 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 0x7fa7bc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c0b46640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400eaa0 tx=0x7fa7b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7b97fa640 1 -- 172.21.15.107:0/3643259913 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400cdb0 con 0x7fa7bc107460 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7b97fa640 1 -- 172.21.15.107:0/3643259913 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7b4005020 con 0x7fa7bc107460 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc1a1c10 con 0x7fa7bc107460 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7b97fa640 1 -- 172.21.15.107:0/3643259913 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40116b0 con 0x7fa7bc107460 2024-04-24T23:45:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.865+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc1a2110 con 0x7fa7bc107460 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.867+0000 7fa7b97fa640 1 -- 172.21.15.107:0/3643259913 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa7b4005560 con 0x7fa7bc107460 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 Processor -- start 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 -- start start 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa04930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d61f962f00 con 0x55d61f9ff440 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55d61f90b750 con 0x55d61fa04fa0 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c539a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55d61f95eec0 con 0x55d61f9ffe30 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7b97fa640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57768/0 (socket says 172.21.15.107:57768) 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4089843498 learned_addr learned my addr 172.21.15.107:0/4089843498 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7c0b46640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa04930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7b97fa640 1 -- 172.21.15.107:0/3643259913 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7b40cf700 con 0x7fa7bc107460 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4089843498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 msgr2=0x55d61fa04930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa04930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4089843498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 msgr2=0x55d61fa07430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.868+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4089843498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d61f6772d0 con 0x55d61f9ff440 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c0b46640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 0x7fa7a80a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa7b409bb20 tx=0x7fa7b40c6720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005ab0 tx=0x7fa7a4005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ae7fed45e8e09d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7b27fc640 1 -- 172.21.15.107:0/4089843498 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4035070 con 0x55d61f9ff440 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7b27fc640 1 -- 172.21.15.107:0/4089843498 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005040 con 0x55d61f9ff440 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7b27fc640 1 -- 172.21.15.107:0/4089843498 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004a80 con 0x55d61f9ff440 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 -- 172.21.15.107:0/4089843498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 msgr2=0x55d61f9ff860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005ab0 tx=0x7fa7a4005a40 comp rx=0 tx=0).stop 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 -- 172.21.15.107:0/4089843498 shutdown_connections 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa04930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4089843498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ff440 0x55d61f9ff860 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 -- 172.21.15.107:0/4089843498 >> 172.21.15.107:0/4089843498 conn(0x55d61f9faab0 msgr2=0x55d61f9faeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 -- 172.21.15.107:0/4089843498 shutdown_connections 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.869+0000 7fa7c539a740 1 -- 172.21.15.107:0/4089843498 wait complete. 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 -- start start 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 Processor -- start 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61fa097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa10d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa07e90 con 0x55d61f9ffe30 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55d61f91ecf0 con 0x55d61fa04fa0 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa08010 con 0x55d61f9ff440 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7c539a740 10 client.0 _ll_register_callbacks cb 0x55d61f91e6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57774/0 (socket says 172.21.15.107:57774) 2024-04-24T23:45:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7bbfff640 1 -- 172.21.15.107:0/3794036173 learned_addr learned my addr 172.21.15.107:0/3794036173 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.870+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61fa097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa10d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 -- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 msgr2=0x55d61fa097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61fa097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 -- 172.21.15.107:0/3794036173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 msgr2=0x55d61fa10d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa10d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 -- 172.21.15.107:0/3794036173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d61f7416b0 con 0x55d61f9ffe30 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa10d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac010820 tx=0x7fa7ac010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bf90475a3506a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00fab0 con 0x55d61f9ffe30 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7ac00fc50 con 0x55d61f9ffe30 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a4005040 con 0x55d61f9ffe30 2024-04-24T23:45:08.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa7ac00fe90 con 0x55d61f9ffe30 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 10 client.15468 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d61fa13de0 con 0x55d61f9ffe30 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005040 con 0x55d61f9ffe30 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa7ac004a00 con 0x55d61f9ffe30 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 10 client.15468 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 10 client.15468 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 20 client.15468 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa16520 0x55d61fa17990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 4 client.15468 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.871+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d61f970440 con 0x55d61fa16520 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.872+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa16520 0x55d61fa17990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.872+0000 7fa79b7fe640 10 client.15468 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.872+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa16520 0x55d61fa17990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c606444472c224ea server_cookie=b55421fa77b7c919 in_seq=0 out_seq=0 2024-04-24T23:45:08.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa79b7fe640 1 -- 172.21.15.107:0/3794036173 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d61f970440 con 0x55d61fa16520 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa79b7fe640 10 client.15468 handle_command_reply: tid=0 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 client.15468 shutdown 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 msgr2=0x55d61fa09cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac010820 tx=0x7fa7ac010ce0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 shutdown_connections 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa16520 0x55d61fa17990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa10d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa79b7fe640 0 client.15468 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa09cf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 --2- 172.21.15.107:0/3794036173 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61fa097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 >> 172.21.15.107:0/3794036173 conn(0x55d61f9faab0 msgr2=0x55d61f9f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 shutdown_connections 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 1 -- 172.21.15.107:0/3794036173 wait complete. 2024-04-24T23:45:08.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.873+0000 7fa7c539a740 20 client.15468 trim_cache size 0 max 16384 2024-04-24T23:45:08.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 Processor -- start 2024-04-24T23:45:08.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 -- start start 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61f9f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa15cd0 con 0x55d61f9ffe30 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa163a0 con 0x55d61fa04fa0 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7c539a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa16a70 con 0x55d61f9ff440 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57780/0 (socket says 172.21.15.107:57780) 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 -- 172.21.15.107:0/1877831101 learned_addr learned my addr 172.21.15.107:0/1877831101 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 -- 172.21.15.107:0/1877831101 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 msgr2=0x55d61f9f9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61f9f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 -- 172.21.15.107:0/1877831101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 msgr2=0x55d61fa07390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.891+0000 7fa7bbfff640 1 -- 172.21.15.107:0/1877831101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d61f9fa5b0 con 0x55d61f9ffe30 2024-04-24T23:45:08.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0058c0 tx=0x7fa7ac00a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f9cfd57b92a722 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7b1ffb640 1 -- 172.21.15.107:0/1877831101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac02e030 con 0x55d61f9ffe30 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7b1ffb640 1 -- 172.21.15.107:0/1877831101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7ac003730 con 0x55d61f9ffe30 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7b1ffb640 1 -- 172.21.15.107:0/1877831101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac005400 con 0x55d61f9ffe30 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 -- 172.21.15.107:0/1877831101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 msgr2=0x55d61fa10cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0058c0 tx=0x7fa7ac00a450 comp rx=0 tx=0).stop 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 -- 172.21.15.107:0/1877831101 shutdown_connections 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61fa04fa0 0x55d61fa07390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61f9ffe30 0x55d61fa10cf0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 --2- 172.21.15.107:0/1877831101 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ff440 0x55d61f9f9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 -- 172.21.15.107:0/1877831101 >> 172.21.15.107:0/1877831101 conn(0x55d61f9faab0 msgr2=0x55d61f9f5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 -- 172.21.15.107:0/1877831101 shutdown_connections 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.892+0000 7fa7c539a740 1 -- 172.21.15.107:0/1877831101 wait complete. 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 -- start start 2024-04-24T23:45:08.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 Processor -- start 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61f9ff440 0x55d61fa0c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa0cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa17810 con 0x55d61fa04fa0 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa17140 con 0x55d61f9ff440 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55d61fa17ee0 con 0x55d61f9ffe30 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7c539a740 10 client.0 _ll_register_callbacks cb 0x7fa7ac010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57782/0 (socket says 172.21.15.107:57782) 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4168024491 learned_addr learned my addr 172.21.15.107:0/4168024491 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 msgr2=0x55d61fa0cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa0cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa0cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4168024491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61f9ff440 msgr2=0x55d61fa0c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61f9ff440 0x55d61fa0c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.893+0000 7fa7bb7fe640 1 -- 172.21.15.107:0/4168024491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d61f9fb780 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7bb7fe640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40314e0 tx=0x7fa7a4005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a20ea1ee551e66a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4035040 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d61f96a790 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7a4004030 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa7a4004230 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 10 client.15474 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d61fa7c910 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d61f96a790 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa7a40035f0 con 0x55d61fa04fa0 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 10 client.15474 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 10 client.15474 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 20 client.15474 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:08.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa811d0 0x55d61fa7fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 4 client.15474 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.894+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "status"}) -- 0x55d61f8e8b80 con 0x55d61fa811d0 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa811d0 0x55d61fa7fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa79b7fe640 10 client.15474 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7bbfff640 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa811d0 0x55d61fa7fee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57842ddd6faefae5 server_cookie=c1189b59689ec948 in_seq=0 out_seq=0 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa79b7fe640 1 -- 172.21.15.107:0/4168024491 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d61f8e8b80 con 0x55d61fa811d0 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa79b7fe640 10 client.15474 handle_command_reply: tid=0 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 client.15474 shutdown 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 msgr2=0x55d61fa0ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40314e0 tx=0x7fa7a4005de0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 shutdown_connections 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55d61fa811d0 0x55d61fa7fee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d61fa04fa0 0x55d61fa0ff50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55d61f9ffe30 0x55d61fa0cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa79b7fe640 0 client.15474 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 --2- 172.21.15.107:0/4168024491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55d61f9ff440 0x55d61fa0c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 >> 172.21.15.107:0/4168024491 conn(0x55d61f9faab0 msgr2=0x55d61f9fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.895+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 shutdown_connections 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c539a740 1 -- 172.21.15.107:0/4168024491 wait complete. 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c539a740 20 client.15474 trim_cache size 0 max 16384 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 3, 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24463, 2024-04-24T23:45:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 111.81545699599999, 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 158.92320345499999 2024-04-24T23:45:08.903 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 msgr2=0x7fa7a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 0x7fa7a80a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa7b409bb20 tx=0x7fa7b40c6720 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 msgr2=0x7fa7bc1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400eaa0 tx=0x7fa7b400ef60 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 shutdown_connections 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7bc107460 0x7fa7bc1006f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa7bc104b30 0x7fa7bc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 --2- 172.21.15.107:0/3643259913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7bc102200 0x7fa7bc101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 >> 172.21.15.107:0/3643259913 conn(0x7fa7bc0fbbf0 msgr2=0x7fa7bc102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 shutdown_connections 2024-04-24T23:45:08.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:08.896+0000 7fa7c25a9640 1 -- 172.21.15.107:0/3643259913 wait complete. 2024-04-24T23:45:08.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-24T23:45:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 -- 172.21.15.107:0/1020259526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 msgr2=0x7f3870108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 --2- 172.21.15.107:0/1020259526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f3870108ac0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3868009ca0 tx=0x7f386802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 -- 172.21.15.107:0/1020259526 shutdown_connections 2024-04-24T23:45:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 --2- 172.21.15.107:0/1020259526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f3870108ac0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 --2- 172.21.15.107:0/1020259526 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 0x7f3870101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 --2- 172.21.15.107:0/1020259526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 0x7f3870100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.005+0000 7f3876e95640 1 -- 172.21.15.107:0/1020259526 >> 172.21.15.107:0/1020259526 conn(0x7f38700fbbf0 msgr2=0x7f38700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- 172.21.15.107:0/1020259526 shutdown_connections 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- 172.21.15.107:0/1020259526 wait complete. 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 Processor -- start 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- start start 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 0x7f3870069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 0x7f387019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f387010c2e0 con 0x7f3870102230 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f387010c160 con 0x7f3870100680 2024-04-24T23:45:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3876e95640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f387010c460 con 0x7f3870101870 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3875432640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3875432640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57806/0 (socket says 172.21.15.107:57806) 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.006+0000 7f3875432640 1 -- 172.21.15.107:0/2908919587 learned_addr learned my addr 172.21.15.107:0/2908919587 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3874c31640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 0x7f3870069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f386ffff640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 0x7f387019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 -- 172.21.15.107:0/2908919587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 msgr2=0x7f387019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 0x7f387019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 -- 172.21.15.107:0/2908919587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 msgr2=0x7f3870069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 0x7f3870069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 -- 172.21.15.107:0/2908919587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387019f6d0 con 0x7f3870102230 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3875432640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3868005e30 tx=0x7f3868031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f386dffb640 1 -- 172.21.15.107:0/2908919587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868003c30 con 0x7f3870102230 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f386dffb640 1 -- 172.21.15.107:0/2908919587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3868003dd0 con 0x7f3870102230 2024-04-24T23:45:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387019f900 con 0x7f3870102230 2024-04-24T23:45:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f386dffb640 1 -- 172.21.15.107:0/2908919587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386802fde0 con 0x7f3870102230 2024-04-24T23:45:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.007+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387019fe50 con 0x7f3870102230 2024-04-24T23:45:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.009+0000 7f386dffb640 1 -- 172.21.15.107:0/2908919587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3868004050 con 0x7f3870102230 2024-04-24T23:45:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 Processor -- start 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 -- start start 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386dffb640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 0x7f38600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7baa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7b05f00 con 0x55e3b7ba2e30 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7aae750 con 0x55e3b7ba7fa0 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3879b9b740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7b01ec0 con 0x55e3b7ba2440 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386dffb640 1 -- 172.21.15.107:0/2908919587 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f38680f09d0 con 0x7f3870102230 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386ffff640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 0x7f38600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:43492/0 (socket says 172.21.15.107:43492) 2024-04-24T23:45:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 -- 172.21.15.107:0/2493356765 learned_addr learned my addr 172.21.15.107:0/2493356765 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386ffff640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57814/0 (socket says 172.21.15.107:57814) 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3875432640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7baa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 -- 172.21.15.107:0/2493356765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 msgr2=0x55e3b7baa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7baa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 -- 172.21.15.107:0/2493356765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 msgr2=0x55e3b7ba7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f3874c31640 1 -- 172.21.15.107:0/2493356765 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3b781a2d0 con 0x55e3b7ba2440 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.010+0000 7f386ffff640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f386ffff640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 0x7f38600a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f385c005fd0 tx=0x7f385c00bac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f3874c31640 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f386400dcb0 tx=0x7f386400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=150a90ddfd2fd168 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f385affd640 1 -- 172.21.15.107:0/2493356765 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864011440 con 0x55e3b7ba2440 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f385affd640 1 -- 172.21.15.107:0/2493356765 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3864011ac0 con 0x55e3b7ba2440 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f385affd640 1 -- 172.21.15.107:0/2493356765 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864010930 con 0x55e3b7ba2440 2024-04-24T23:45:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f3879b9b740 1 -- 172.21.15.107:0/2493356765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 msgr2=0x55e3b7ba2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f386400dcb0 tx=0x7f386400f730 comp rx=0 tx=0).stop 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.011+0000 7f3879b9b740 1 -- 172.21.15.107:0/2493356765 shutdown_connections 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7baa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7ba7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2493356765 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7ba2860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- 172.21.15.107:0/2493356765 >> 172.21.15.107:0/2493356765 conn(0x55e3b7b9dab0 msgr2=0x55e3b7b9deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- 172.21.15.107:0/2493356765 shutdown_connections 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- 172.21.15.107:0/2493356765 wait complete. 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- start start 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 Processor -- start 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bacd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7baaf50 con 0x55e3b7ba2440 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7ac1cf0 con 0x55e3b7ba2e30 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bab0d0 con 0x55e3b7ba7fa0 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3879b9b740 10 client.0 _ll_register_callbacks cb 0x55e3b7ac16e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3874c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3874c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57824/0 (socket says 172.21.15.107:57824) 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3874c31640 1 -- 172.21.15.107:0/2090986595 learned_addr learned my addr 172.21.15.107:0/2090986595 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.012+0000 7f3875432640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f386ffff640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bacd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 -- 172.21.15.107:0/2090986595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 msgr2=0x55e3b7bb3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 -- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 msgr2=0x55e3b7bacd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bacd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 -- 172.21.15.107:0/2090986595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3b78e46b0 con 0x55e3b7ba2440 2024-04-24T23:45:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3875432640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3874c31640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3864000c00 tx=0x7f386400d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e4cf1cf6237148 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386402b060 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3864012d00 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3864011ac0 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3864032050 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3879b9b740 10 client.15486 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3b7bb6d90 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3864011ac0 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3864004750 con 0x55e3b7ba2440 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3879b9b740 10 client.15486 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.013+0000 7f3879b9b740 10 client.15486 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f3879b9b740 20 client.15486 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7bb9020 0x55e3b7bba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f3879b9b740 4 client.15486 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3b7b13440 con 0x55e3b7bb9020 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f386ffff640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7bb9020 0x55e3b7bba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f386ffff640 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7bb9020 0x55e3b7bba490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c9623c4b7b436f62 server_cookie=62ec92fda79be37e in_seq=0 out_seq=0 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.014+0000 7f384ffff640 10 client.15486 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f384ffff640 1 -- 172.21.15.107:0/2090986595 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3b7b13440 con 0x55e3b7bb9020 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f384ffff640 10 client.15486 handle_command_reply: tid=0 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 client.15486 shutdown 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 msgr2=0x55e3b7bac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3864000c00 tx=0x7f386400d9f0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 shutdown_connections 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7bb9020 0x55e3b7bba490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bacd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f384ffff640 0 client.15486 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2090986595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7bac830 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 >> 172.21.15.107:0/2090986595 conn(0x55e3b7b9dab0 msgr2=0x55e3b7b9cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 shutdown_connections 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 1 -- 172.21.15.107:0/2090986595 wait complete. 2024-04-24T23:45:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.015+0000 7f3879b9b740 20 client.15486 trim_cache size 0 max 16384 2024-04-24T23:45:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 Processor -- start 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 -- start start 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7b9c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bb87d0 con 0x55e3b7ba2440 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bb8ea0 con 0x55e3b7ba2e30 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3879b9b740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bb9570 con 0x55e3b7ba7fa0 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3874c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3874c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57826/0 (socket says 172.21.15.107:57826) 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.033+0000 7f3874c31640 1 -- 172.21.15.107:0/2763464691 learned_addr learned my addr 172.21.15.107:0/2763464691 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 -- 172.21.15.107:0/2763464691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 msgr2=0x55e3b7b9c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3875432640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7b9c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7b9c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f386ffff640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 -- 172.21.15.107:0/2763464691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 msgr2=0x55e3b7bb3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 -- 172.21.15.107:0/2763464691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3b7ba1fa0 con 0x55e3b7ba2440 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f386ffff640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f3874c31640 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f38640048c0 tx=0x7f38640048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75da757fdb483822 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f385a7fc640 1 -- 172.21.15.107:0/2763464691 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386402b030 con 0x55e3b7ba2440 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f385a7fc640 1 -- 172.21.15.107:0/2763464691 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3864011440 con 0x55e3b7ba2440 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.034+0000 7f385a7fc640 1 -- 172.21.15.107:0/2763464691 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864010580 con 0x55e3b7ba2440 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- 172.21.15.107:0/2763464691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 msgr2=0x55e3b7b98690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f38640048c0 tx=0x7f38640048f0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- 172.21.15.107:0/2763464691 shutdown_connections 2024-04-24T23:45:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7b9c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- 172.21.15.107:0/2763464691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2440 0x55e3b7b98690 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- 172.21.15.107:0/2763464691 >> 172.21.15.107:0/2763464691 conn(0x55e3b7b9dab0 msgr2=0x55e3b7b9de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- 172.21.15.107:0/2763464691 shutdown_connections 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- 172.21.15.107:0/2763464691 wait complete. 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- start start 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 Processor -- start 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7baff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bba310 con 0x55e3b7ba2e30 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bb9c40 con 0x55e3b7ba7fa0 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e3b7bba9e0 con 0x55e3b7ba2440 2024-04-24T23:45:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.035+0000 7f3879b9b740 10 client.0 _ll_register_callbacks cb 0x7f3868009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57828/0 (socket says 172.21.15.107:57828) 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 -- 172.21.15.107:0/322958076 learned_addr learned my addr 172.21.15.107:0/322958076 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 -- 172.21.15.107:0/322958076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 msgr2=0x55e3b7baff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7baff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 -- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 msgr2=0x55e3b7bb2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 -- 172.21.15.107:0/322958076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3b7bb3510 con 0x55e3b7ba2e30 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f386ffff640 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f385c00b870 tx=0x7f385c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6016f1e3846b0112 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f384ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c02f060 con 0x55e3b7ba2e30 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f385c004730 con 0x55e3b7ba2e30 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3864011440 con 0x55e3b7ba2e30 2024-04-24T23:45:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f385c036050 con 0x55e3b7ba2e30 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f3879b9b740 10 client.15492 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3b7c1f4a0 con 0x55e3b7ba2e30 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.036+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3864011440 con 0x55e3b7ba2e30 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f385c029370 con 0x55e3b7ba2e30 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 10 client.15492 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 10 client.15492 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 20 client.15492 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7c24150 0x55e3b7c22e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 4 client.15492 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "status"}) -- 0x55e3b7b0d790 con 0x55e3b7c24150 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3874c31640 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7c24150 0x55e3b7c22e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3874c31640 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7c24150 0x55e3b7c22e60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b677bf3505782cad server_cookie=ed0bfd0927f684e3 in_seq=0 out_seq=0 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f384ffff640 10 client.15492 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:09.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f384ffff640 1 -- 172.21.15.107:0/322958076 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e3b7b0d790 con 0x55e3b7c24150 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f384ffff640 10 client.15492 handle_command_reply: tid=0 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 1 client.15492 shutdown 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.037+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 msgr2=0x55e3b7bb0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f385c00b870 tx=0x7f385c01a000 comp rx=0 tx=0).stop 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 shutdown_connections 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55e3b7c24150 0x55e3b7c22e60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e3b7ba7fa0 0x55e3b7bb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f384ffff640 0 client.15492 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3b7ba2e30 0x55e3b7bb0460 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 --2- 172.21.15.107:0/322958076 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e3b7ba2440 0x55e3b7baff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 >> 172.21.15.107:0/322958076 conn(0x55e3b7b9dab0 msgr2=0x55e3b7b98690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 shutdown_connections 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 1 -- 172.21.15.107:0/322958076 wait complete. 2024-04-24T23:45:09.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3879b9b740 20 client.15492 trim_cache size 0 max 16384 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24464, 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 112.965118109, 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-04-24T23:45:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:45:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 156.19517359100001 2024-04-24T23:45:09.046 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 msgr2=0x7f38600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 0x7f38600a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f385c005fd0 tx=0x7f385c00bac0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 msgr2=0x7f387019edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3868005e30 tx=0x7f3868031ee0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 shutdown_connections 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f38600a2b90 0x7f38600a5050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3870102230 0x7f387019edc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3870101870 0x7f387019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 --2- 172.21.15.107:0/2908919587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3870100680 0x7f3870069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 >> 172.21.15.107:0/2908919587 conn(0x7f38700fbbf0 msgr2=0x7f3870104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 shutdown_connections 2024-04-24T23:45:09.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.038+0000 7f3876e95640 1 -- 172.21.15.107:0/2908919587 wait complete. 2024-04-24T23:45:09.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/670419971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58102230 msgr2=0x7f0c58108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/670419971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58102230 0x7f0c58108ac0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009900 tx=0x7f0c5002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/670419971 shutdown_connections 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/670419971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58102230 0x7f0c58108ac0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/670419971 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58101870 0x7f0c58101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/670419971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 0x7f0c58100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/670419971 >> 172.21.15.107:0/670419971 conn(0x7f0c580fbbf0 msgr2=0x7f0c580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/670419971 shutdown_connections 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.149+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/670419971 wait complete. 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 Processor -- start 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 -- start start 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 0x7f0c58196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58102230 0x7f0c5819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c300 con 0x7f0c58101870 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c180 con 0x7f0c58100680 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5f9ed640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c480 con 0x7f0c58102230 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57846/0 (socket says 172.21.15.107:57846) 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 -- 172.21.15.107:0/1416926679 learned_addr learned my addr 172.21.15.107:0/1416926679 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 -- 172.21.15.107:0/1416926679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58102230 msgr2=0x7f0c5819df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58102230 0x7f0c5819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5d789640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 0x7f0c58196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58102230 0x7f0c5819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 -- 172.21.15.107:0/1416926679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 msgr2=0x7f0c58196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 0x7f0c58196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 -- 172.21.15.107:0/1416926679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5819e590 con 0x7f0c58101870 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.150+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009860 tx=0x7f0c48009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.151+0000 7f0c4e7fc640 1 -- 172.21.15.107:0/1416926679 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48005020 con 0x7f0c58101870 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.151+0000 7f0c4e7fc640 1 -- 172.21.15.107:0/1416926679 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c480047a0 con 0x7f0c58101870 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.151+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c58068930 con 0x7f0c58101870 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.151+0000 7f0c4e7fc640 1 -- 172.21.15.107:0/1416926679 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48007ba0 con 0x7f0c58101870 2024-04-24T23:45:09.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.151+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c58068e30 con 0x7f0c58101870 2024-04-24T23:45:09.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c4e7fc640 1 -- 172.21.15.107:0/1416926679 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0c48004950 con 0x7f0c58101870 2024-04-24T23:45:09.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 Processor -- start 2024-04-24T23:45:09.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 -- start start 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65882930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562d657e0f00 con 0x562d65882fa0 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562d65789750 con 0x562d6587de30 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c627cf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562d657dcec0 con 0x562d6587d440 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c4e7fc640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 0x7f0c440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.153+0000 7f0c5cf88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57854/0 (socket says 172.21.15.107:57854) 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 -- 172.21.15.107:0/2362564636 learned_addr learned my addr 172.21.15.107:0/2362564636 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65882930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 0x7f0c440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c4e7fc640 1 -- 172.21.15.107:0/1416926679 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0c480d0a00 con 0x7f0c58101870 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 -- 172.21.15.107:0/2362564636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 msgr2=0x562d6587d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 -- 172.21.15.107:0/2362564636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 msgr2=0x562d65882930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65882930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 -- 172.21.15.107:0/2362564636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d654f52d0 con 0x562d65882fa0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65882930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 0x7f0c440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x562d6587e690 tx=0x7f0c50031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c4809d050 tx=0x7f0c4809cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85b3d205aaacbb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c437fe640 1 -- 172.21.15.107:0/2362564636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4809d590 con 0x562d65882fa0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c437fe640 1 -- 172.21.15.107:0/2362564636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c480a1240 con 0x562d65882fa0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.154+0000 7f0c437fe640 1 -- 172.21.15.107:0/2362564636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4809c2e0 con 0x562d65882fa0 2024-04-24T23:45:09.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- 172.21.15.107:0/2362564636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 msgr2=0x562d65885430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c4809d050 tx=0x7f0c4809cb00 comp rx=0 tx=0).stop 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- 172.21.15.107:0/2362564636 shutdown_connections 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d65885430 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65882930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2362564636 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- 172.21.15.107:0/2362564636 >> 172.21.15.107:0/2362564636 conn(0x562d65878ab0 msgr2=0x562d65878eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- 172.21.15.107:0/2362564636 shutdown_connections 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- 172.21.15.107:0/2362564636 wait complete. 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 -- start start 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.155+0000 7f0c627cf740 1 Processor -- start 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65887d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 0x562d6588ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562d65885e90 con 0x562d6587d440 2024-04-24T23:45:09.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562d6579ccf0 con 0x562d6587de30 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562d65886010 con 0x562d65882fa0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 10 client.0 _ll_register_callbacks cb 0x562d6579c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57864/0 (socket says 172.21.15.107:57864) 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 -- 172.21.15.107:0/934838677 learned_addr learned my addr 172.21.15.107:0/934838677 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 0x562d6588ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65887d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 -- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 msgr2=0x562d6588ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 0x562d6588ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 -- 172.21.15.107:0/934838677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 msgr2=0x562d65887d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65887d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 -- 172.21.15.107:0/934838677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d655bf6b0 con 0x562d6587d440 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 0x562d6588ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c5d789640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c54010a50 tx=0x7f0c54010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a6d216a9bc419a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5401d060 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0c5400eb60 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.156+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c480a1240 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c54024050 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 10 client.15504 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562d65891e60 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c480a1240 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c5400d3a0 con 0x562d6587d440 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 10 client.15504 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 10 client.15504 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 20 client.15504 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65894550 0x562d658959c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 4 client.15504 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:09.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d657ee440 con 0x562d65894550 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.157+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65894550 0x562d658959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.158+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65894550 0x562d658959c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d031412f629e5e7 server_cookie=e63a78e9f8ca6fef in_seq=0 out_seq=0 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.158+0000 7f0c40ff9640 10 client.15504 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c40ff9640 1 -- 172.21.15.107:0/934838677 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d657ee440 con 0x562d65894550 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c40ff9640 10 client.15504 handle_command_reply: tid=0 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 client.15504 shutdown 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 msgr2=0x562d658877e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c54010a50 tx=0x7f0c54010f10 comp rx=0 tx=0).stop 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 shutdown_connections 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d65882fa0 0x562d6588ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65894550 0x562d658959c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d65887d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c40ff9640 0 client.15504 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:09.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 --2- 172.21.15.107:0/934838677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d6587d440 0x562d658877e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 >> 172.21.15.107:0/934838677 conn(0x562d65878ab0 msgr2=0x562d65877ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 shutdown_connections 2024-04-24T23:45:09.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 1 -- 172.21.15.107:0/934838677 wait complete. 2024-04-24T23:45:09.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.159+0000 7f0c627cf740 20 client.15504 trim_cache size 0 max 16384 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.176+0000 7f0c627cf740 1 Processor -- start 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.176+0000 7f0c627cf740 1 -- start start 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d6587bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562d65893d00 con 0x562d65882fa0 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562d658943d0 con 0x562d6587de30 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c627cf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562d65894aa0 con 0x562d6587d440 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57868/0 (socket says 172.21.15.107:57868) 2024-04-24T23:45:09.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 -- 172.21.15.107:0/4107268332 learned_addr learned my addr 172.21.15.107:0/4107268332 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 -- 172.21.15.107:0/4107268332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 msgr2=0x562d6587b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 -- 172.21.15.107:0/4107268332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 msgr2=0x562d6587bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d6587bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 -- 172.21.15.107:0/4107268332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d6587e210 con 0x562d65882fa0 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c50031b50 tx=0x7f0c50005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc34db28b3a82b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c42ffd640 1 -- 172.21.15.107:0/4107268332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50035040 con 0x562d65882fa0 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c42ffd640 1 -- 172.21.15.107:0/4107268332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c5004f070 con 0x562d65882fa0 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.177+0000 7f0c42ffd640 1 -- 172.21.15.107:0/4107268332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c500035f0 con 0x562d65882fa0 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- 172.21.15.107:0/4107268332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 msgr2=0x562d6587c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c50031b50 tx=0x7f0c50005fd0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- 172.21.15.107:0/4107268332 shutdown_connections 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d6587c210 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d6587bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 --2- 172.21.15.107:0/4107268332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d6587b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- 172.21.15.107:0/4107268332 >> 172.21.15.107:0/4107268332 conn(0x562d65878ab0 msgr2=0x562d6587f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- 172.21.15.107:0/4107268332 shutdown_connections 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- 172.21.15.107:0/4107268332 wait complete. 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 -- start start 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.178+0000 7f0c627cf740 1 Processor -- start 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d658f86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d658fac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562d65895840 con 0x562d65882fa0 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562d65895170 con 0x562d6587de30 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562d65895f10 con 0x562d6587d440 2024-04-24T23:45:09.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 10 client.0 _ll_register_callbacks cb 0x7f0c54010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57880/0 (socket says 172.21.15.107:57880) 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 -- 172.21.15.107:0/2738155128 learned_addr learned my addr 172.21.15.107:0/2738155128 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d658f86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d658fac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 -- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 msgr2=0x562d658f86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d658f86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 -- 172.21.15.107:0/2738155128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 msgr2=0x562d658fac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d658fac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 -- 172.21.15.107:0/2738155128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d658fd9d0 con 0x562d65882fa0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d658f86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5cf88640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d658fac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c5df8a640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002a930 tx=0x7f0c50003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a200ebd350d443 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50058070 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0c50003e20 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c5004f070 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c50064050 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 10 client.15510 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.179+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562d658fecd0 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c5004f070 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c5003a030 con 0x562d65882fa0 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 10 client.15510 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 10 client.15510 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 20 client.15510 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65903460 0x562d65902170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 4 client.15510 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:45:09.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "status"}) -- 0x562d657e8790 con 0x562d65903460 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65903460 0x562d65902170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c40ff9640 10 client.15510 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.180+0000 7f0c5d789640 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65903460 0x562d65902170 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27319ea9b88c173 server_cookie=e1fc262f5a70d086 in_seq=0 out_seq=0 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c40ff9640 1 -- 172.21.15.107:0/2738155128 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x562d657e8790 con 0x562d65903460 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c40ff9640 10 client.15510 handle_command_reply: tid=0 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 client.15510 shutdown 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 msgr2=0x562d658fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002a930 tx=0x7f0c50003820 comp rx=0 tx=0).stop 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 shutdown_connections 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d65882fa0 0x562d658fd1a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562d65903460 0x562d65902170 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562d6587de30 0x562d658fac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 --2- 172.21.15.107:0/2738155128 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562d6587d440 0x562d658f86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c40ff9640 0 client.15510 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 >> 172.21.15.107:0/2738155128 conn(0x562d65878ab0 msgr2=0x562d65877250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 shutdown_connections 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 1 -- 172.21.15.107:0/2738155128 wait complete. 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c627cf740 20 client.15510 trim_cache size 0 max 16384 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24493, 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:45:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 114.11818394700001, 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 153.46985783 2024-04-24T23:45:09.189 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 msgr2=0x7f0c440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 0x7f0c440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x562d6587e690 tx=0x7f0c50031400 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 msgr2=0x7f0c58196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009860 tx=0x7f0c48009d20 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 shutdown_connections 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0c440a2b90 0x7f0c440a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0c58102230 0x7f0c5819df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c58101870 0x7f0c58196eb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 --2- 172.21.15.107:0/1416926679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c58100680 0x7f0c58196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 >> 172.21.15.107:0/1416926679 conn(0x7f0c580fbbf0 msgr2=0x7f0c58104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 shutdown_connections 2024-04-24T23:45:09.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.181+0000 7f0c5f9ed640 1 -- 172.21.15.107:0/1416926679 wait complete. 2024-04-24T23:45:09.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:09 smithi107 ceph-mon[29118]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:09.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:09 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1453710900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.191 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:09 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3110166309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 -- 172.21.15.107:0/2888054587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 msgr2=0x7fddbc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 --2- 172.21.15.107:0/2888054587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc100aa0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fddac009900 tx=0x7fddac02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 -- 172.21.15.107:0/2888054587 shutdown_connections 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 --2- 172.21.15.107:0/2888054587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 --2- 172.21.15.107:0/2888054587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 --2- 172.21.15.107:0/2888054587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc100aa0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 -- 172.21.15.107:0/2888054587 >> 172.21.15.107:0/2888054587 conn(0x7fddbc0fbbf0 msgr2=0x7fddbc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 -- 172.21.15.107:0/2888054587 shutdown_connections 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 -- 172.21.15.107:0/2888054587 wait complete. 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.292+0000 7fddc40d6640 1 Processor -- start 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- start start 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c350 con 0x7fddbc100680 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c1d0 con 0x7fddbc101870 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c4d0 con 0x7fddbc102230 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57904/0 (socket says 172.21.15.107:57904) 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 -- 172.21.15.107:0/3039376733 learned_addr learned my addr 172.21.15.107:0/3039376733 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc2673640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1671640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 -- 172.21.15.107:0/3039376733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 msgr2=0x7fddbc19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 -- 172.21.15.107:0/3039376733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 msgr2=0x7fddbc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 -- 172.21.15.107:0/3039376733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddbc19e5b0 con 0x7fddbc100680 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1671640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc2673640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fddac031440 tx=0x7fddac032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddb2ffd640 1 -- 172.21.15.107:0/3039376733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac03f030 con 0x7fddbc100680 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddb2ffd640 1 -- 172.21.15.107:0/3039376733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddac003ce0 con 0x7fddbc100680 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddb2ffd640 1 -- 172.21.15.107:0/3039376733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac035070 con 0x7fddbc100680 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddbc19e840 con 0x7fddbc100680 2024-04-24T23:45:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.293+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddbc19ed80 con 0x7fddbc100680 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.295+0000 7fddb2ffd640 1 -- 172.21.15.107:0/3039376733 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fddac042030 con 0x7fddbc100680 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 Processor -- start 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 -- start start 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddb2ffd640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 0x7fdda80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae4572650 0x561ae4572a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4577b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae44d3f00 con 0x561ae45781b0 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561ae447c750 con 0x561ae4573040 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc6de1740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561ae44cfec0 con 0x561ae4572650 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc2673640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 0x7fdda80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc2673640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddb2ffd640 1 -- 172.21.15.107:0/3039376733 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fddac0efa20 con 0x7fddbc100680 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc2673640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57908/0 (socket says 172.21.15.107:57908) 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc2673640 1 -- 172.21.15.107:0/3285010139 learned_addr learned my addr 172.21.15.107:0/3285010139 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.296+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae4572650 0x561ae4572a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc1671640 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4577b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 -- 172.21.15.107:0/3285010139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae4572650 msgr2=0x561ae4572a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae4572650 0x561ae4572a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 -- 172.21.15.107:0/3285010139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 msgr2=0x561ae4577b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4577b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 -- 172.21.15.107:0/3285010139 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae41e82d0 con 0x561ae45781b0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 0x7fdda80a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fddb40099e0 tx=0x7fddb4009ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fddc2673640 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fddb4000b60 tx=0x7fddb401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7b80b0e4919a42 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fdda7fff640 1 -- 172.21.15.107:0/3285010139 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4024070 con 0x561ae45781b0 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fdda7fff640 1 -- 172.21.15.107:0/3285010139 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddb401fc00 con 0x561ae45781b0 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.297+0000 7fdda7fff640 1 -- 172.21.15.107:0/3285010139 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb401e650 con 0x561ae45781b0 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- 172.21.15.107:0/3285010139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 msgr2=0x561ae457a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fddb4000b60 tx=0x7fddb401c000 comp rx=0 tx=0).stop 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- 172.21.15.107:0/3285010139 shutdown_connections 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae45781b0 0x561ae457a640 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4577b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3285010139 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae4572650 0x561ae4572a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- 172.21.15.107:0/3285010139 >> 172.21.15.107:0/3285010139 conn(0x561ae456bab0 msgr2=0x561ae456beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- 172.21.15.107:0/3285010139 shutdown_connections 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- 172.21.15.107:0/3285010139 wait complete. 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 -- start start 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.298+0000 7fddc6de1740 1 Processor -- start 2024-04-24T23:45:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae457cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4583fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae457b100 con 0x561ae4572650 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561ae448fcf0 con 0x561ae4573040 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561ae457b280 con 0x561ae45781b0 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 10 client.0 _ll_register_callbacks cb 0x561ae448f6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57920/0 (socket says 172.21.15.107:57920) 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 -- 172.21.15.107:0/3791503504 learned_addr learned my addr 172.21.15.107:0/3791503504 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1671640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae457cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc2673640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4583fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 -- 172.21.15.107:0/3791503504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 msgr2=0x561ae4583fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4583fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 -- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 msgr2=0x561ae457cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae457cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 -- 172.21.15.107:0/3791503504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae42b26b0 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc1e72640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fddac0ed1a0 tx=0x7fddac0bb9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992c24fdddb6699 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fdda57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac049040 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fddac03bd30 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddb401fc00 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddac0bbeb0 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 10 client.15522 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.299+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ae4586fc0 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddb401fc00 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddac0b8520 con 0x561ae4572650 2024-04-24T23:45:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 10 client.15522 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 10 client.15522 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 20 client.15522 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae4589690 0x561ae458ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 4 client.15522 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ae44e1440 con 0x561ae4589690 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.300+0000 7fddc1671640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae4589690 0x561ae458ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fdda57fa640 10 client.15522 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc1671640 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae4589690 0x561ae458ab00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d44a5a9769ef55a server_cookie=2eefafe13849ad7b in_seq=0 out_seq=0 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fdda57fa640 1 -- 172.21.15.107:0/3791503504 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ae44e1440 con 0x561ae4589690 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fdda57fa640 10 client.15522 handle_command_reply: tid=0 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc6de1740 1 client.15522 shutdown 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 msgr2=0x561ae457ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fddac0ed1a0 tx=0x7fddac0bb9f0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 shutdown_connections 2024-04-24T23:45:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.301+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae4589690 0x561ae458ab00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4583fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae457cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 --2- 172.21.15.107:0/3791503504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae457ca60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fdda57fa640 0 client.15522 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 >> 172.21.15.107:0/3791503504 conn(0x561ae456bab0 msgr2=0x561ae456aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 shutdown_connections 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 1 -- 172.21.15.107:0/3791503504 wait complete. 2024-04-24T23:45:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.302+0000 7fddc6de1740 20 client.15522 trim_cache size 0 max 16384 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.319+0000 7fddc6de1740 1 Processor -- start 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.319+0000 7fddc6de1740 1 -- start start 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.319+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4574400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4574940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc6de1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae4588e40 con 0x561ae4572650 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc6de1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561ae4589510 con 0x561ae4573040 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc6de1740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561ae4589be0 con 0x561ae45781b0 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57926/0 (socket says 172.21.15.107:57926) 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 -- 172.21.15.107:0/1578977803 learned_addr learned my addr 172.21.15.107:0/1578977803 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 -- 172.21.15.107:0/1578977803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 msgr2=0x561ae4574940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4574940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 -- 172.21.15.107:0/1578977803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 msgr2=0x561ae4574400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4574400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 -- 172.21.15.107:0/1578977803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae45776f0 con 0x561ae4572650 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fddc1e72640 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fddac0ed4c0 tx=0x7fddac03b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf3dc35be3069a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fdda77fe640 1 -- 172.21.15.107:0/1578977803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac0bb810 con 0x561ae4572650 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fdda77fe640 1 -- 172.21.15.107:0/1578977803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddac0c4b20 con 0x561ae4572650 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.320+0000 7fdda77fe640 1 -- 172.21.15.107:0/1578977803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac0c48f0 con 0x561ae4572650 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- 172.21.15.107:0/1578977803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 msgr2=0x561ae4577a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fddac0ed4c0 tx=0x7fddac03b800 comp rx=0 tx=0).stop 2024-04-24T23:45:09.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- 172.21.15.107:0/1578977803 shutdown_connections 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae4574940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4573040 0x561ae4574400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- 172.21.15.107:0/1578977803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4572650 0x561ae4577a70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- 172.21.15.107:0/1578977803 >> 172.21.15.107:0/1578977803 conn(0x561ae456bab0 msgr2=0x561ae456be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- 172.21.15.107:0/1578977803 shutdown_connections 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- 172.21.15.107:0/1578977803 wait complete. 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- start start 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 Processor -- start 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4572650 0x561ae4571670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae456ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae458a980 con 0x561ae4573040 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561ae458a2b0 con 0x561ae4572650 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561ae458b050 con 0x561ae45781b0 2024-04-24T23:45:09.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc6de1740 10 client.0 _ll_register_callbacks cb 0x7fddac0b8240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.321+0000 7fddc1671640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57938/0 (socket says 172.21.15.107:57938) 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 -- 172.21.15.107:0/2139207664 learned_addr learned my addr 172.21.15.107:0/2139207664 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc2673640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae456ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1e72640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4572650 0x561ae4571670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 -- 172.21.15.107:0/2139207664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 msgr2=0x561ae456ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae456ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 -- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4572650 msgr2=0x561ae4571670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4572650 0x561ae4571670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 -- 172.21.15.107:0/2139207664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae45706e0 con 0x561ae4573040 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc1671640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x561ae4573ab0 tx=0x7fddb8010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cd49d06e1d61d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:09.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8020030 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fddb801c6f0 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddac0c4b20 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddb802a050 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc6de1740 10 client.15528 fetch_fsmap learned FSMap version 28 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ae45f2310 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddac0c4b20 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.322+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddb8025700 con 0x561ae4573040 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 10 client.15528 fetch_fsmap finished waiting for FSMap version 28 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 10 client.15528 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 20 client.15528 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae45f6ee0 0x561ae45f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 4 client.15528 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "status"}) -- 0x561ae44db790 con 0x561ae45f6ee0 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc1e72640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae45f6ee0 0x561ae45f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:09.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc1e72640 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae45f6ee0 0x561ae45f39f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b19a0accaa1087fa server_cookie=f15d0c836250b23a in_seq=0 out_seq=0 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fdda57fa640 10 client.15528 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fdda57fa640 1 -- 172.21.15.107:0/2139207664 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561ae44db790 con 0x561ae45f6ee0 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fdda57fa640 10 client.15528 handle_command_reply: tid=0 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 1 client.15528 shutdown 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 msgr2=0x561ae4571bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.323+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x561ae4573ab0 tx=0x7fddb8010af0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 shutdown_connections 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x561ae45f6ee0 0x561ae45f39f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561ae45781b0 0x561ae456ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae4573040 0x561ae4571bb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fdda57fa640 0 client.15528 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 --2- 172.21.15.107:0/2139207664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561ae4572650 0x561ae4571670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 >> 172.21.15.107:0/2139207664 conn(0x561ae456bab0 msgr2=0x561ae4574400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 shutdown_connections 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 1 -- 172.21.15.107:0/2139207664 wait complete. 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc6de1740 20 client.15528 trim_cache size 0 max 16384 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "16fa2ae8-0293-11ef-bc93-c7b262605968", 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24506, 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-04-24T23:45:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 138.04131966700001, 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 144.91583876499999 2024-04-24T23:45:09.332 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 msgr2=0x7fdda80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 0x7fdda80a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fddb40099e0 tx=0x7fddb4009ed0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 msgr2=0x7fddbc196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fddac031440 tx=0x7fddac032ce0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 shutdown_connections 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdda80a2c10 0x7fdda80a50d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fddbc102230 0x7fddbc19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddbc101870 0x7fddbc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 --2- 172.21.15.107:0/3039376733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddbc100680 0x7fddbc196990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 >> 172.21.15.107:0/3039376733 conn(0x7fddbc0fbbf0 msgr2=0x7fddbc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 shutdown_connections 2024-04-24T23:45:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:09.324+0000 7fddc40d6640 1 -- 172.21.15.107:0/3039376733 wait complete. 2024-04-24T23:45:09.335 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T23:45:09.344 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T23:45:09.344 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T23:45:09.345 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T23:45:09.345 INFO:teuthology.run_tasks:Running task kclient... 2024-04-24T23:45:09.358 INFO:tasks.kclient:Mounting kernel clients... 2024-04-24T23:45:09.358 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-24T23:45:09.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:09.358 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-04-24T23:45:09.373 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-24T23:45:09.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:09.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:45:09.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:09.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2024-04-24T23:45:09.531 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-24T23:45:09.532 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:45:09.532 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:09.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-24T23:45:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:09 smithi195 ceph-mon[33019]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:09 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1453710900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:09 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3110166309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:09 smithi170 ceph-mon[32714]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:09 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1453710900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:09 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3110166309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:09.771 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:45:11.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:11 smithi107 ceph-mon[29118]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:11.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 -- 172.21.15.107:0/965840359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70381082b0 msgr2=0x7f7038108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 --2- 172.21.15.107:0/965840359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70381082b0 0x7f7038108690 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7028009990 tx=0x7f7028009ca0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 -- 172.21.15.107:0/965840359 shutdown_connections 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 --2- 172.21.15.107:0/965840359 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 0x7f703810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 --2- 172.21.15.107:0/965840359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70381082b0 0x7f7038108690 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 --2- 172.21.15.107:0/965840359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7038069f40 0x7f703806a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 -- 172.21.15.107:0/965840359 >> 172.21.15.107:0/965840359 conn(0x7f70380780e0 msgr2=0x7f70380784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 -- 172.21.15.107:0/965840359 shutdown_connections 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.442+0000 7f7040374700 1 -- 172.21.15.107:0/965840359 wait complete. 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 Processor -- start 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 -- start start 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 0x7f7038101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70381082b0 0x7f7038101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7038112560 con 0x7f7038069f40 2024-04-24T23:45:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f70381123e0 con 0x7f70381082b0 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f7040374700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f70381126e0 con 0x7f703806a8e0 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f703e110700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f703e110700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57946/0 (socket says 172.21.15.107:57946) 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f703e110700 1 -- 172.21.15.107:0/2876560829 learned_addr learned my addr 172.21.15.107:0/2876560829 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f703d90f700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 0x7f7038101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.443+0000 7f703e911700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70381082b0 0x7f7038101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 -- 172.21.15.107:0/2876560829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 msgr2=0x7f7038101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 0x7f7038101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 -- 172.21.15.107:0/2876560829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70381082b0 msgr2=0x7f7038101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70381082b0 0x7f7038101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 -- 172.21.15.107:0/2876560829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70381a6f20 con 0x7f7038069f40 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f703e110700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7028005910 tx=0x7f70280047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7028027070 con 0x7f7038069f40 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70381a71b0 con 0x7f7038069f40 2024-04-24T23:45:11.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7028003e40 con 0x7f7038069f40 2024-04-24T23:45:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7028022d00 con 0x7f7038069f40 2024-04-24T23:45:11.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.444+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70381a7660 con 0x7f7038069f40 2024-04-24T23:45:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.445+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7038068f20 con 0x7f7038069f40 2024-04-24T23:45:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.446+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f702802a070 con 0x7f7038069f40 2024-04-24T23:45:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.447+0000 7f702f7fe700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 0x7f70240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.447+0000 7f703d90f700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 0x7f70240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.447+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f70280d0190 con 0x7f7038069f40 2024-04-24T23:45:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.447+0000 7f703d90f700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 0x7f70240a50e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f70380786e0 tx=0x7f703400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:11.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.451+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f702805e740 con 0x7f7038069f40 2024-04-24T23:45:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:11 smithi195 ceph-mon[33019]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:11 smithi170 ceph-mon[32714]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.691+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f703802db30 con 0x7f7038069f40 2024-04-24T23:45:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.691+0000 7f702f7fe700 1 -- 172.21.15.107:0/2876560829 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f7028013070 con 0x7f7038069f40 2024-04-24T23:45:11.692 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:45:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 msgr2=0x7f70240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 0x7f70240a50e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f70380786e0 tx=0x7f703400b410 comp rx=0 tx=0).stop 2024-04-24T23:45:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 msgr2=0x7f7038101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7028005910 tx=0x7f70280047b0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 shutdown_connections 2024-04-24T23:45:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70240a2c20 0x7f70240a50e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70381082b0 0x7f7038101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7038069f40 0x7f7038101200 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 --2- 172.21.15.107:0/2876560829 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f703806a8e0 0x7f7038101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 >> 172.21.15.107:0/2876560829 conn(0x7f70380780e0 msgr2=0x7f70380ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 shutdown_connections 2024-04-24T23:45:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:11.693+0000 7f7040374700 1 -- 172.21.15.107:0/2876560829 wait complete. 2024-04-24T23:45:12.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:12 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2876560829' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:45:12.419 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:45:12.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:12.420 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-04-24T23:45:12.436 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:45:12.437 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42096sec preferred_lft 42096sec 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::6f34:b7ae:a569:134c/64 scope link noprefixroute 2024-04-24T23:45:12.438 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:45:12.438 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:45:12.438 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:45:12.438 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:45:12.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:45:12.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-04-24T23:45:12.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-04-24T23:45:12.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:45:12.439 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:45:12.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:12.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:12 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2876560829' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:45:12.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:12.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:12.577 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:45:12.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:12 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2876560829' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T23:45:12.599 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-04-24T23:45:12.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:12.601 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-04-24T23:45:12.657 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-04-24T23:45:12.657 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:45:12.657 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T23:45:12.657 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T23:45:12.657 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:45:12.657 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:45:12.658 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:45:12.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T23:45:12.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T23:45:12.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T23:45:12.658 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:45:12.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:12.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:12.782 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-04-24T23:45:12.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:12.797 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-04-24T23:45:12.852 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:45:12.852 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:45:12.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:45:12.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T23:45:12.853 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:45:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:12.967 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T23:45:12.967 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:45:12.967 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:45:12.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T23:45:12.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:45:12.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T23:45:12.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T23:45:12.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T23:45:12.968 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:45:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:13.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.076 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:45:13.076 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-04-24T23:45:13.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-04-24T23:45:13.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:45:13.076 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-04-24T23:45:13.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:13.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.151 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:45:13.151 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:13.166 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:45:13.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:45:13.166 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:13.221 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-24T23:45:13.221 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-24T23:45:13.221 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/14a26539-10ae-41c8-bb82-2677aaedd4e9 2024-04-24T23:45:13.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:13.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/14a26539-10ae-41c8-bb82-2677aaedd4e9 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-24T23:45:13.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:13 smithi107 ceph-mon[29118]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:13.302 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-24T23:45:13.302 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-24T23:45:13.302 INFO:teuthology.orchestra.run.smithi107.stdout:invalid new device string format 2024-04-24T23:45:13.422 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-24T23:45:13.422 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-24T23:45:13.422 INFO:teuthology.orchestra.run.smithi107.stdout:invalid new device string format 2024-04-24T23:45:13.423 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.170:3300,172.21.15.195:3300" 2024-04-24T23:45:13.423 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.107:3300,172.21.15.170:3300,172.21.15.195:3300:/volumes/qa/sv_1/14a26539-10ae-41c8-bb82-2677aaedd4e9 2024-04-24T23:45:13.423 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=16fa2ae8-0293-11ef-bc93-c7b262605968" will pass to kernel 2024-04-24T23:45:13.424 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-24T23:45:13.424 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2024-04-24T23:45:13.424 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-24T23:45:13.424 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2024-04-24T23:45:13.424 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-24T23:45:13.425 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-24T23:45:13.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:13.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:13.491 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:45:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:13 smithi195 ceph-mon[33019]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:13.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:45:13.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:13 smithi170 ceph-mon[32714]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:13.586 INFO:teuthology.orchestra.run.smithi107.stdout:client25043 2024-04-24T23:45:13.586 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:45:13.586 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:45:13.603 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:45:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:45:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.698 INFO:teuthology.orchestra.run.smithi107.stdout:client25043 2024-04-24T23:45:13.698 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:45:13.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:45:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.753 INFO:teuthology.orchestra.run.smithi107.stdout:client25043 2024-04-24T23:45:13.753 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968.client25043/status') 2024-04-24T23:45:13.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:45:13.809 INFO:teuthology.orchestra.run.smithi107.stdout:instance: client.25043 (3)172.21.15.107:0/2448935596 2024-04-24T23:45:13.809 INFO:teuthology.orchestra.run.smithi107.stdout:blocklisted: false 2024-04-24T23:45:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records in 2024-04-24T23:45:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records out 2024-04-24T23:45:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:72 bytes copied, 2.2597e-05 s, 3.2 MB/s 2024-04-24T23:45:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:45:13.817 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T23:45:13.828 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T23:45:13.828 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:45:13.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-24T23:45:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 -- 172.21.15.107:0/875895232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18100070 msgr2=0x7fdb18100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 --2- 172.21.15.107:0/875895232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18100070 0x7fdb18100490 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009900 tx=0x7fdb1402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 -- 172.21.15.107:0/875895232 shutdown_connections 2024-04-24T23:45:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 --2- 172.21.15.107:0/875895232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb18101370 0x7fdb181017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 --2- 172.21.15.107:0/875895232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb181009d0 0x7fdb18100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 --2- 172.21.15.107:0/875895232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18100070 0x7fdb18100490 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 -- 172.21.15.107:0/875895232 >> 172.21.15.107:0/875895232 conn(0x7fdb180fbc30 msgr2=0x7fdb180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 -- 172.21.15.107:0/875895232 shutdown_connections 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.928+0000 7fdb207ee640 1 -- 172.21.15.107:0/875895232 wait complete. 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 Processor -- start 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 -- start start 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb18100070 0x7fdb18199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 0x7fdb1819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1810f600 con 0x7fdb18101370 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1810f480 con 0x7fdb181009d0 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1810f780 con 0x7fdb18100070 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:13.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57958/0 (socket says 172.21.15.107:57958) 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 -- 172.21.15.107:0/1757774681 learned_addr learned my addr 172.21.15.107:0/1757774681 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1e58a640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb18100070 0x7fdb18199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1dd89640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 0x7fdb1819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 -- 172.21.15.107:0/1757774681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb18100070 msgr2=0x7fdb18199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb18100070 0x7fdb18199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 -- 172.21.15.107:0/1757774681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 msgr2=0x7fdb1819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 0x7fdb1819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 -- 172.21.15.107:0/1757774681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb181a17a0 con 0x7fdb18101370 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1dd89640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 0x7fdb1819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb1ed8b640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000cce0 tx=0x7fdb1000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1000f450 con 0x7fdb18101370 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb181a1a90 con 0x7fdb18101370 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.929+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb1000fad0 con 0x7fdb18101370 2024-04-24T23:45:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.930+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10018c10 con 0x7fdb18101370 2024-04-24T23:45:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.930+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb181a1f90 con 0x7fdb18101370 2024-04-24T23:45:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.930+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb1810ae90 con 0x7fdb18101370 2024-04-24T23:45:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.931+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdb1000f5f0 con 0x7fdb18101370 2024-04-24T23:45:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.932+0000 7fdb0f7fe640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 0x7fdb040a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:13.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.932+0000 7fdb1e58a640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 0x7fdb040a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:13.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.932+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdb100cfb00 con 0x7fdb18101370 2024-04-24T23:45:13.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.933+0000 7fdb1e58a640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 0x7fdb040a50f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdb140314e0 tx=0x7fdb14000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:13.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:13.936+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdb10014030 con 0x7fdb18101370 2024-04-24T23:45:14.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:14.156+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fdb1802d170 con 0x7fdb18101370 2024-04-24T23:45:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.133+0000 7fdb0f7fe640 1 -- 172.21.15.107:0/1757774681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7fdb1009c690 con 0x7fdb18101370 2024-04-24T23:45:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 msgr2=0x7fdb040a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 0x7fdb040a50f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdb140314e0 tx=0x7fdb14000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 msgr2=0x7fdb181a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000cce0 tx=0x7fdb1000a620 comp rx=0 tx=0).stop 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 shutdown_connections 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdb040a2c30 0x7fdb040a50f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb18101370 0x7fdb181a1130 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb181009d0 0x7fdb1819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 --2- 172.21.15.107:0/1757774681 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdb18100070 0x7fdb18199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.136+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 >> 172.21.15.107:0/1757774681 conn(0x7fdb180fbc30 msgr2=0x7fdb180fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.137+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 shutdown_connections 2024-04-24T23:45:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.137+0000 7fdb207ee640 1 -- 172.21.15.107:0/1757774681 wait complete. 2024-04-24T23:45:15.150 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-24T23:45:15.162 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-24T23:45:15.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 -- 172.21.15.107:0/4164108235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8101033f0 msgr2=0x7ff810109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 --2- 172.21.15.107:0/4164108235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8101033f0 0x7ff810109c80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff800009930 tx=0x7ff80002a420 comp rx=0 tx=0).stop 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 -- 172.21.15.107:0/4164108235 shutdown_connections 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 --2- 172.21.15.107:0/4164108235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8101033f0 0x7ff810109c80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 --2- 172.21.15.107:0/4164108235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff810102a30 0x7ff810102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 --2- 172.21.15.107:0/4164108235 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 0x7ff8100ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 -- 172.21.15.107:0/4164108235 >> 172.21.15.107:0/4164108235 conn(0x7ff8100fb3b0 msgr2=0x7ff8100fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 -- 172.21.15.107:0/4164108235 shutdown_connections 2024-04-24T23:45:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.284+0000 7ff81577e640 1 -- 172.21.15.107:0/4164108235 wait complete. 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 Processor -- start 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 -- start start 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 0x7ff810197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8101033f0 0x7ff81019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff81010d430 con 0x7ff810102a30 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff81010d2b0 con 0x7ff8101033f0 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff81577e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff81010d5b0 con 0x7ff8100ffa60 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff80e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff80e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57976/0 (socket says 172.21.15.107:57976) 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff80e7fc640 1 -- 172.21.15.107:0/1159093250 learned_addr learned my addr 172.21.15.107:0/1159093250 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff80f7fe640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8101033f0 0x7ff81019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.285+0000 7ff80effd640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 0x7ff810197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 -- 172.21.15.107:0/1159093250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 msgr2=0x7ff810197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 0x7ff810197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 -- 172.21.15.107:0/1159093250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8101033f0 msgr2=0x7ff81019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8101033f0 0x7ff81019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 -- 172.21.15.107:0/1159093250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81019f690 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff80e7fc640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff80400eab0 tx=0x7ff80400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80400ce40 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff804011950 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81019f980 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804010ae0 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.286+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81019fed0 con 0x7ff810102a30 2024-04-24T23:45:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.287+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff810108b60 con 0x7ff810102a30 2024-04-24T23:45:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.288+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff804011b00 con 0x7ff810102a30 2024-04-24T23:45:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.289+0000 7ff7f7fff640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 0x7ff7f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.289+0000 7ff80effd640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 0x7ff7f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.289+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff8040cfa30 con 0x7ff810102a30 2024-04-24T23:45:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.289+0000 7ff80effd640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 0x7ff7f80a5050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff808005fd0 tx=0x7ff80800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.293+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff804016030 con 0x7ff810102a30 2024-04-24T23:45:15.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.475+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff810062370 con 0x7ff810102a30 2024-04-24T23:45:15.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.476+0000 7ff7f7fff640 1 -- 172.21.15.107:0/1159093250 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7ff80409c5c0 con 0x7ff810102a30 2024-04-24T23:45:15.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:15.477 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 msgr2=0x7ff7f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 0x7ff7f80a5050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff808005fd0 tx=0x7ff80800b390 comp rx=0 tx=0).stop 2024-04-24T23:45:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 msgr2=0x7ff810197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff80400eab0 tx=0x7ff80400ef70 comp rx=0 tx=0).stop 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 shutdown_connections 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7f80a2b90 0x7ff7f80a5050 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8101033f0 0x7ff81019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff810102a30 0x7ff810197fb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 --2- 172.21.15.107:0/1159093250 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff8100ffa60 0x7ff810197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.478+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 >> 172.21.15.107:0/1159093250 conn(0x7ff8100fb3b0 msgr2=0x7ff810105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.479+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 shutdown_connections 2024-04-24T23:45:15.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.479+0000 7ff81577e640 1 -- 172.21.15.107:0/1159093250 wait complete. 2024-04-24T23:45:15.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:15 smithi107 ceph-mon[29118]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:15.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:15 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-24T23:45:15.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:45:15.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:45:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:15 smithi195 ceph-mon[33019]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:15 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-24T23:45:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:15 smithi170 ceph-mon[32714]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:15 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-24T23:45:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.590+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/176805311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 msgr2=0x7fd1ec100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.590+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/176805311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100d70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009900 tx=0x7fd1e002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/176805311 shutdown_connections 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/176805311 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 0x7fd1ec108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/176805311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100d70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/176805311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 0x7fd1ec1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/176805311 >> 172.21.15.107:0/176805311 conn(0x7fd1ec0fb380 msgr2=0x7fd1ec0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/176805311 shutdown_connections 2024-04-24T23:45:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/176805311 wait complete. 2024-04-24T23:45:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 Processor -- start 2024-04-24T23:45:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- start start 2024-04-24T23:45:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 0x7fd1ec19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 0x7fd1ec19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec10c380 con 0x7fd1ec1008f0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec10c200 con 0x7fd1ec101ca0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.591+0000 7fd1f0bb7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec10c500 con 0x7fd1ec1012b0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58002/0 (socket says 172.21.15.107:58002) 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 -- 172.21.15.107:0/918897161 learned_addr learned my addr 172.21.15.107:0/918897161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ead9d640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 0x7fd1ec19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1e9d9b640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 0x7fd1ec19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 -- 172.21.15.107:0/918897161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 msgr2=0x7fd1ec19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 0x7fd1ec19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 -- 172.21.15.107:0/918897161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 msgr2=0x7fd1ec19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 0x7fd1ec19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 -- 172.21.15.107:0/918897161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ec19f750 con 0x7fd1ec1008f0 2024-04-24T23:45:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1ea59c640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d770 tx=0x7fd1dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc017070 con 0x7fd1ec1008f0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc009c30 con 0x7fd1ec1008f0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ec19fa40 con 0x7fd1ec1008f0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc003dd0 con 0x7fd1ec1008f0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.592+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ec19ff90 con 0x7fd1ec1008f0 2024-04-24T23:45:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.593+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1ec107b50 con 0x7fd1ec1008f0 2024-04-24T23:45:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.594+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd1dc003fb0 con 0x7fd1ec1008f0 2024-04-24T23:45:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.595+0000 7fd1db7fe640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 0x7fd1d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.595+0000 7fd1e9d9b640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 0x7fd1d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.595+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd1dc0ceda0 con 0x7fd1ec1008f0 2024-04-24T23:45:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.595+0000 7fd1e9d9b640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 0x7fd1d40a5050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0006010 tx=0x7fd1e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.599+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1dc09b930 con 0x7fd1ec1008f0 2024-04-24T23:45:15.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.775+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1ec062340 con 0x7fd1ec1008f0 2024-04-24T23:45:15.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.776+0000 7fd1db7fe640 1 -- 172.21.15.107:0/918897161 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd1dc009de0 con 0x7fd1ec1008f0 2024-04-24T23:45:15.776 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:45:15.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 msgr2=0x7fd1d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 0x7fd1d40a5050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0006010 tx=0x7fd1e0000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 msgr2=0x7fd1ec100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d770 tx=0x7fd1dc00dc30 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 shutdown_connections 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd1d40a2b90 0x7fd1d40a5050 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1ec101ca0 0x7fd1ec19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd1ec1012b0 0x7fd1ec19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 --2- 172.21.15.107:0/918897161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1ec1008f0 0x7fd1ec100370 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 >> 172.21.15.107:0/918897161 conn(0x7fd1ec0fb380 msgr2=0x7fd1ec104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 shutdown_connections 2024-04-24T23:45:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.778+0000 7fd1f0bb7640 1 -- 172.21.15.107:0/918897161 wait complete. 2024-04-24T23:45:15.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:45:15.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- 172.21.15.107:0/3803260398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090104b30 msgr2=0x7f7090106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 --2- 172.21.15.107:0/3803260398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090104b30 0x7f7090106f20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7084009900 tx=0x7f708402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- 172.21.15.107:0/3803260398 shutdown_connections 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 --2- 172.21.15.107:0/3803260398 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 0x7f70901098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 --2- 172.21.15.107:0/3803260398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090104b30 0x7f7090106f20 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 --2- 172.21.15.107:0/3803260398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090102200 0x7f70901045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- 172.21.15.107:0/3803260398 >> 172.21.15.107:0/3803260398 conn(0x7f70900fbbf0 msgr2=0x7f70900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- 172.21.15.107:0/3803260398 shutdown_connections 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- 172.21.15.107:0/3803260398 wait complete. 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 Processor -- start 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.890+0000 7f7095272640 1 -- start start 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090104b30 0x7f7090198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 0x7f709019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d450 con 0x7f7090102200 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d2d0 con 0x7f7090104b30 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d5d0 con 0x7f7090107460 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58014/0 (socket says 172.21.15.107:58014) 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 -- 172.21.15.107:0/1908615551 learned_addr learned my addr 172.21.15.107:0/1908615551 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708e7fc640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090104b30 0x7f7090198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708f7fe640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 0x7f709019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 -- 172.21.15.107:0/1908615551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 msgr2=0x7f709019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 0x7f709019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 -- 172.21.15.107:0/1908615551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090104b30 msgr2=0x7f7090198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090104b30 0x7f7090198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 -- 172.21.15.107:0/1908615551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709019f720 con 0x7f7090102200 2024-04-24T23:45:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f708effd640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f708000d950 tx=0x7f708000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080017070 con 0x7f7090102200 2024-04-24T23:45:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7080009c30 con 0x7f7090102200 2024-04-24T23:45:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709019fa10 con 0x7f7090102200 2024-04-24T23:45:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708000f6b0 con 0x7f7090102200 2024-04-24T23:45:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.891+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709019ffb0 con 0x7f7090102200 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.892+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70900fd6e0 con 0x7f7090102200 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.893+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f70800109d0 con 0x7f7090102200 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.894+0000 7f7077fff640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 0x7f70780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.894+0000 7f708e7fc640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 0x7f70780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.894+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f70800cf150 con 0x7f7090102200 2024-04-24T23:45:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.894+0000 7f708e7fc640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 0x7f70780a5050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7084006010 tx=0x7f7084000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:15.898+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7080009de0 con 0x7f7090102200 2024-04-24T23:45:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.074+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7090062370 con 0x7f7090102200 2024-04-24T23:45:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.075+0000 7f7077fff640 1 -- 172.21.15.107:0/1908615551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f708009bce0 con 0x7f7090102200 2024-04-24T23:45:16.075 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 msgr2=0x7f70780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 0x7f70780a5050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7084006010 tx=0x7f7084000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 msgr2=0x7f7090197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f708000d950 tx=0x7f708000de10 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 shutdown_connections 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70780a2b90 0x7f70780a5050 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7090107460 0x7f709019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7090104b30 0x7f7090198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 --2- 172.21.15.107:0/1908615551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7090102200 0x7f7090197b00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 >> 172.21.15.107:0/1908615551 conn(0x7f70900fbbf0 msgr2=0x7f7090102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 shutdown_connections 2024-04-24T23:45:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.077+0000 7f7095272640 1 -- 172.21.15.107:0/1908615551 wait complete. 2024-04-24T23:45:16.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 -- 172.21.15.107:0/3067191436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10101870 msgr2=0x7fde10101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 --2- 172.21.15.107:0/3067191436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10101870 0x7fde10101cf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fde04009900 tx=0x7fde0402a3f0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 -- 172.21.15.107:0/3067191436 shutdown_connections 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 --2- 172.21.15.107:0/3067191436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10102230 0x7fde10108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 --2- 172.21.15.107:0/3067191436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10101870 0x7fde10101cf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 --2- 172.21.15.107:0/3067191436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10100680 0x7fde10100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 -- 172.21.15.107:0/3067191436 >> 172.21.15.107:0/3067191436 conn(0x7fde100fbbf0 msgr2=0x7fde100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 -- 172.21.15.107:0/3067191436 shutdown_connections 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.191+0000 7fde16eba640 1 -- 172.21.15.107:0/3067191436 wait complete. 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 Processor -- start 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 -- start start 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10100680 0x7fde101968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10101870 0x7fde10196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010c2e0 con 0x7fde10102230 2024-04-24T23:45:16.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010c160 con 0x7fde10101870 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde16eba640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010c460 con 0x7fde10100680 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde15457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde15457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58030/0 (socket says 172.21.15.107:58030) 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde15457640 1 -- 172.21.15.107:0/4221546274 learned_addr learned my addr 172.21.15.107:0/4221546274 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde14c56640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10100680 0x7fde101968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.192+0000 7fde0ffff640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10101870 0x7fde10196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 -- 172.21.15.107:0/4221546274 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10100680 msgr2=0x7fde101968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10100680 0x7fde101968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 -- 172.21.15.107:0/4221546274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10101870 msgr2=0x7fde10196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10101870 0x7fde10196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 -- 172.21.15.107:0/4221546274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1019e510 con 0x7fde10102230 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde15457640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fde0800cce0 tx=0x7fde0800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800f450 con 0x7fde10102230 2024-04-24T23:45:16.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0800fad0 con 0x7fde10102230 2024-04-24T23:45:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1019e800 con 0x7fde10102230 2024-04-24T23:45:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08018c10 con 0x7fde10102230 2024-04-24T23:45:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.193+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1019ed20 con 0x7fde10102230 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.194+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde10107940 con 0x7fde10102230 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.195+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fde0800f5f0 con 0x7fde10102230 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.195+0000 7fde0dffb640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 0x7fddf80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.196+0000 7fde14c56640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 0x7fddf80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.196+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fde080cfa60 con 0x7fde10102230 2024-04-24T23:45:16.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.196+0000 7fde14c56640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 0x7fddf80a50d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde101016d0 tx=0x7fde00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.200+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde08014030 con 0x7fde10102230 2024-04-24T23:45:16.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.376+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde1002d170 con 0x7fde10102230 2024-04-24T23:45:16.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.377+0000 7fde0dffb640 1 -- 172.21.15.107:0/4221546274 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fde0809c5f0 con 0x7fde10102230 2024-04-24T23:45:16.377 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:45:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-24T23:45:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:16 smithi107 ceph-mon[29118]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:45:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1159093250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:16.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/918897161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:16 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1908615551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 msgr2=0x7fddf80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 0x7fddf80a50d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde101016d0 tx=0x7fde00005f90 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 msgr2=0x7fde1019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fde0800cce0 tx=0x7fde0800a620 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 shutdown_connections 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fddf80a2c10 0x7fddf80a50d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde10102230 0x7fde1019dea0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde10101870 0x7fde10196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 --2- 172.21.15.107:0/4221546274 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fde10100680 0x7fde101968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 >> 172.21.15.107:0/4221546274 conn(0x7fde100fbbf0 msgr2=0x7fde10104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 shutdown_connections 2024-04-24T23:45:16.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.379+0000 7fde16eba640 1 -- 172.21.15.107:0/4221546274 wait complete. 2024-04-24T23:45:16.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 -- 172.21.15.107:0/1154990757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c102230 msgr2=0x7f0f6c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 --2- 172.21.15.107:0/1154990757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c102230 0x7f0f6c108ac0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009ca0 tx=0x7f0f6402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 -- 172.21.15.107:0/1154990757 shutdown_connections 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 --2- 172.21.15.107:0/1154990757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c102230 0x7f0f6c108ac0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 --2- 172.21.15.107:0/1154990757 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c101870 0x7f0f6c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 --2- 172.21.15.107:0/1154990757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c100680 0x7f0f6c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 -- 172.21.15.107:0/1154990757 >> 172.21.15.107:0/1154990757 conn(0x7f0f6c0fbbf0 msgr2=0x7f0f6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 -- 172.21.15.107:0/1154990757 shutdown_connections 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.491+0000 7f0f71af5640 1 -- 172.21.15.107:0/1154990757 wait complete. 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 Processor -- start 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- start start 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 0x7f0f6c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 0x7f0f6c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10c300 con 0x7f0f6c100680 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10c180 con 0x7f0f6c101870 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10c480 con 0x7f0f6c102230 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42820/0 (socket says 172.21.15.107:42820) 2024-04-24T23:45:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 -- 172.21.15.107:0/3358792606 learned_addr learned my addr 172.21.15.107:0/3358792606 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f63fff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 0x7f0f6c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6bfff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 0x7f0f6c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 -- 172.21.15.107:0/3358792606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 msgr2=0x7f0f6c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 0x7f0f6c19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 -- 172.21.15.107:0/3358792606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 msgr2=0x7f0f6c19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 0x7f0f6c19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 -- 172.21.15.107:0/3358792606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c19f6b0 con 0x7f0f6c100680 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f63fff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 0x7f0f6c19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6bfff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 0x7f0f6c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f6b7fe640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00e9c0 tx=0x7f0f5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c00cd80 con 0x7f0f6c100680 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f5c0035f0 con 0x7f0f6c100680 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6c19f9a0 con 0x7f0f6c100680 2024-04-24T23:45:16.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c010690 con 0x7f0f6c100680 2024-04-24T23:45:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.492+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f6c19fef0 con 0x7f0f6c100680 2024-04-24T23:45:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.494+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f6c107a70 con 0x7f0f6c100680 2024-04-24T23:45:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.495+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0f5c0037a0 con 0x7f0f6c100680 2024-04-24T23:45:16.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.495+0000 7f0f697fa640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 0x7f0f580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.495+0000 7f0f63fff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 0x7f0f580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.496+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0f5c0cf950 con 0x7f0f6c100680 2024-04-24T23:45:16.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.496+0000 7f0f63fff640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 0x7f0f580a5050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0f54005fd0 tx=0x7f0f54005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.500+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f5c09c4e0 con 0x7f0f6c100680 2024-04-24T23:45:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-24T23:45:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:16 smithi195 ceph-mon[33019]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:45:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1159093250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/918897161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:16 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1908615551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1757774681' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-24T23:45:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:16 smithi170 ceph-mon[32714]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-24T23:45:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1159093250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:16.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/918897161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:16 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1908615551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.680+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f6c062340 con 0x7f0f6c100680 2024-04-24T23:45:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.680+0000 7f0f697fa640 1 -- 172.21.15.107:0/3358792606 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f0f5c09bde0 con 0x7f0f6c100680 2024-04-24T23:45:16.680 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:16.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 msgr2=0x7f0f580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 0x7f0f580a5050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0f54005fd0 tx=0x7f0f54005ea0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 msgr2=0x7f0f6c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00e9c0 tx=0x7f0f5c00ee80 comp rx=0 tx=0).stop 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 shutdown_connections 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0f580a2b90 0x7f0f580a5050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0f6c102230 0x7f0f6c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f6c101870 0x7f0f6c19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 --2- 172.21.15.107:0/3358792606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f6c100680 0x7f0f6c069460 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 >> 172.21.15.107:0/3358792606 conn(0x7f0f6c0fbbf0 msgr2=0x7f0f6c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 shutdown_connections 2024-04-24T23:45:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.682+0000 7f0f71af5640 1 -- 172.21.15.107:0/3358792606 wait complete. 2024-04-24T23:45:16.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 -- 172.21.15.107:0/1634633830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0101870 msgr2=0x7f4bc0101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 --2- 172.21.15.107:0/1634633830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0101870 0x7f4bc0101cf0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009900 tx=0x7f4bb402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:16.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 -- 172.21.15.107:0/1634633830 shutdown_connections 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 --2- 172.21.15.107:0/1634633830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 --2- 172.21.15.107:0/1634633830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0101870 0x7f4bc0101cf0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 --2- 172.21.15.107:0/1634633830 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bc0100680 0x7f4bc0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 -- 172.21.15.107:0/1634633830 >> 172.21.15.107:0/1634633830 conn(0x7f4bc00fbbf0 msgr2=0x7f4bc00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 -- 172.21.15.107:0/1634633830 shutdown_connections 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.792+0000 7f4bc682c640 1 -- 172.21.15.107:0/1634633830 wait complete. 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 Processor -- start 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 -- start start 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bc0101870 0x7f4bc0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc010c2e0 con 0x7f4bc0100680 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc010c160 con 0x7f4bc0102230 2024-04-24T23:45:16.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bc682c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc010c460 con 0x7f4bc0101870 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42832/0 (socket says 172.21.15.107:42832) 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.793+0000 7f4bbffff640 1 -- 172.21.15.107:0/3218678385 learned_addr learned my addr 172.21.15.107:0/3218678385 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bc4dc9640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 -- 172.21.15.107:0/3218678385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bc0101870 msgr2=0x7f4bc0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bc0101870 0x7f4bc0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 -- 172.21.15.107:0/3218678385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 msgr2=0x7f4bc019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 -- 172.21.15.107:0/3218678385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc019f660 con 0x7f4bc0100680 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bc4dc9640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbffff640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000d950 tx=0x7f4bb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0017020 con 0x7f4bc0100680 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb0009c30 con 0x7f4bc0100680 2024-04-24T23:45:16.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc019f950 con 0x7f4bc0100680 2024-04-24T23:45:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000f720 con 0x7f4bc0100680 2024-04-24T23:45:16.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.794+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc019fea0 con 0x7f4bc0100680 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.795+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc01a0130 con 0x7f4bc0100680 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.796+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4bb001a020 con 0x7f4bc0100680 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.796+0000 7f4bbd7fa640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 0x7f4ba80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.797+0000 7f4bbf7fe640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 0x7f4ba80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.797+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4bb00cf390 con 0x7f4bc0100680 2024-04-24T23:45:16.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.797+0000 7f4bbf7fe640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 0x7f4ba80a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40096c0 tx=0x7f4bb4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:16.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.801+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4bb0021020 con 0x7f4bc0100680 2024-04-24T23:45:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.977+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bc01a0300 con 0x7f4bc0100680 2024-04-24T23:45:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.978+0000 7f4bbd7fa640 1 -- 172.21.15.107:0/3218678385 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7f4bb0009de0 con 0x7f4bc0100680 2024-04-24T23:45:16.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:16.978 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 msgr2=0x7f4ba80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 0x7f4ba80a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40096c0 tx=0x7f4bb4031400 comp rx=0 tx=0).stop 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 msgr2=0x7f4bc0197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000d950 tx=0x7f4bb000de10 comp rx=0 tx=0).stop 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 shutdown_connections 2024-04-24T23:45:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4ba80a2b90 0x7f4ba80a5050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc0102230 0x7f4bc019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bc0101870 0x7f4bc0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 --2- 172.21.15.107:0/3218678385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bc0100680 0x7f4bc0197a40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 >> 172.21.15.107:0/3218678385 conn(0x7f4bc00fbbf0 msgr2=0x7f4bc00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 shutdown_connections 2024-04-24T23:45:16.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:16.980+0000 7f4bc682c640 1 -- 172.21.15.107:0/3218678385 wait complete. 2024-04-24T23:45:16.992 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:45:16.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:45:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 -- 172.21.15.107:0/1425946674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10102990 msgr2=0x7faa10102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1425946674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10102990 0x7faa10102e10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faa00009900 tx=0x7faa0002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 -- 172.21.15.107:0/1425946674 shutdown_connections 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1425946674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa10103350 0x7faa10109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1425946674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10102990 0x7faa10102e10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1425946674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa101017a0 0x7faa10101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 -- 172.21.15.107:0/1425946674 >> 172.21.15.107:0/1425946674 conn(0x7faa100fcd10 msgr2=0x7faa100ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 -- 172.21.15.107:0/1425946674 shutdown_connections 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.093+0000 7faa17ce7640 1 -- 172.21.15.107:0/1425946674 wait complete. 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 Processor -- start 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 -- start start 2024-04-24T23:45:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa101017a0 0x7faa10197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 0x7faa10197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d400 con 0x7faa10103350 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d280 con 0x7faa101017a0 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa17ce7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d580 con 0x7faa10102990 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42840/0 (socket says 172.21.15.107:42840) 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 -- 172.21.15.107:0/1517468161 learned_addr learned my addr 172.21.15.107:0/1517468161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa15282640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 0x7faa10197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 -- 172.21.15.107:0/1517468161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 msgr2=0x7faa10197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 0x7faa10197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 -- 172.21.15.107:0/1517468161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa101017a0 msgr2=0x7faa10197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa15a83640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa101017a0 0x7faa10197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa101017a0 0x7faa10197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.094+0000 7faa16284640 1 -- 172.21.15.107:0/1517468161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1019f630 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa15282640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 0x7faa10197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa16284640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faa0800cce0 tx=0x7faa0800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0800f450 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0800fad0 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1019f920 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08018c10 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.095+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa1019fdf0 con 0x7faa10103350 2024-04-24T23:45:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.096+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa10108ac0 con 0x7faa10103350 2024-04-24T23:45:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.097+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7faa0800f5f0 con 0x7faa10103350 2024-04-24T23:45:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.097+0000 7faa06ffd640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 0x7fa9fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.097+0000 7faa15a83640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 0x7fa9fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.098+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faa080cfb00 con 0x7faa10103350 2024-04-24T23:45:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.098+0000 7faa15a83640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 0x7fa9fc0a5050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005fd0 tx=0x7faa0c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.102+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa08014030 con 0x7faa10103350 2024-04-24T23:45:17.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.281+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa10062340 con 0x7faa10103350 2024-04-24T23:45:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.282+0000 7faa06ffd640 1 -- 172.21.15.107:0/1517468161 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7faa0809c690 con 0x7faa10103350 2024-04-24T23:45:17.282 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:45:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 msgr2=0x7fa9fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 0x7fa9fc0a5050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005fd0 tx=0x7faa0c005c70 comp rx=0 tx=0).stop 2024-04-24T23:45:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 msgr2=0x7faa1019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faa0800cce0 tx=0x7faa0800a620 comp rx=0 tx=0).stop 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 shutdown_connections 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa9fc0a2b90 0x7fa9fc0a5050 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa10103350 0x7faa1019efc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7faa10102990 0x7faa10197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 --2- 172.21.15.107:0/1517468161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa101017a0 0x7faa10197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 >> 172.21.15.107:0/1517468161 conn(0x7faa100fcd10 msgr2=0x7faa10105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 shutdown_connections 2024-04-24T23:45:17.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.284+0000 7faa17ce7640 1 -- 172.21.15.107:0/1517468161 wait complete. 2024-04-24T23:45:17.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:17 smithi107 ceph-mon[29118]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-24T23:45:17.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/4221546274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3358792606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:17 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3218678385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:17.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 -- 172.21.15.107:0/2712677186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 msgr2=0x7f488c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 --2- 172.21.15.107:0/2712677186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c102e00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4874009900 tx=0x7f487402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 -- 172.21.15.107:0/2712677186 shutdown_connections 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 --2- 172.21.15.107:0/2712677186 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 0x7f488c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 --2- 172.21.15.107:0/2712677186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c102e00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 --2- 172.21.15.107:0/2712677186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 0x7f488c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 -- 172.21.15.107:0/2712677186 >> 172.21.15.107:0/2712677186 conn(0x7f488c0fcd20 msgr2=0x7f488c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 -- 172.21.15.107:0/2712677186 shutdown_connections 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 -- 172.21.15.107:0/2712677186 wait complete. 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.397+0000 7f489162a640 1 Processor -- start 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 -- start start 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 0x7f488c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 0x7f488c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10d410 con 0x7f488c102980 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10d290 con 0x7f488c101790 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10d590 con 0x7f488c103340 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42852/0 (socket says 172.21.15.107:42852) 2024-04-24T23:45:17.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 -- 172.21.15.107:0/1251108641 learned_addr learned my addr 172.21.15.107:0/1251108641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f488b7fe640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 0x7f488c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f488affd640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 0x7f488c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 -- 172.21.15.107:0/1251108641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 msgr2=0x7f488c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 0x7f488c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 -- 172.21.15.107:0/1251108641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 msgr2=0x7f488c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 0x7f488c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 -- 172.21.15.107:0/1251108641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c19f6d0 con 0x7f488c102980 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f48827fc640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4874009230 tx=0x7f4874004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487403f030 con 0x7f488c102980 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4874003ce0 con 0x7f488c102980 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c19f960 con 0x7f488c102980 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.398+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874035070 con 0x7f488c102980 2024-04-24T23:45:17.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.399+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c19fe50 con 0x7f488c102980 2024-04-24T23:45:17.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.400+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c108b30 con 0x7f488c102980 2024-04-24T23:45:17.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.400+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4874042030 con 0x7f488c102980 2024-04-24T23:45:17.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.401+0000 7f4888ff9640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 0x7f48780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.401+0000 7f488affd640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 0x7f48780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.401+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f48740efd50 con 0x7f488c102980 2024-04-24T23:45:17.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.402+0000 7f488affd640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 0x7f48780a5050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f488c1027e0 tx=0x7f487c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.406+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f48740f3030 con 0x7f488c102980 2024-04-24T23:45:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:17 smithi195 ceph-mon[33019]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-24T23:45:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:17 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/4221546274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:17 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3358792606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:17 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3218678385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:17.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.582+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f488c062340 con 0x7f488c102980 2024-04-24T23:45:17.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.583+0000 7f4888ff9640 1 -- 172.21.15.107:0/1251108641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f48740dae20 con 0x7f488c102980 2024-04-24T23:45:17.583 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 msgr2=0x7f48780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 0x7f48780a5050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f488c1027e0 tx=0x7f487c00b390 comp rx=0 tx=0).stop 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 msgr2=0x7f488c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4874009230 tx=0x7f4874004360 comp rx=0 tx=0).stop 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 shutdown_connections 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48780a2b90 0x7f48780a5050 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f488c103340 0x7f488c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f488c102980 0x7f488c197ff0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 --2- 172.21.15.107:0/1251108641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f488c101790 0x7f488c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 >> 172.21.15.107:0/1251108641 conn(0x7f488c0fcd20 msgr2=0x7f488c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 shutdown_connections 2024-04-24T23:45:17.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.585+0000 7f489162a640 1 -- 172.21.15.107:0/1251108641 wait complete. 2024-04-24T23:45:17.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:17 smithi170 ceph-mon[32714]: pgmap v367: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-24T23:45:17.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:17 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/4221546274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:17 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3358792606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:17.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:17 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3218678385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:17.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:45:17.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 -- 172.21.15.107:0/3082276648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c101790 msgr2=0x7f710c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 --2- 172.21.15.107:0/3082276648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c101790 0x7f710c101bb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009900 tx=0x7f70fc02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 -- 172.21.15.107:0/3082276648 shutdown_connections 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 --2- 172.21.15.107:0/3082276648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c103340 0x7f710c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 --2- 172.21.15.107:0/3082276648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c102980 0x7f710c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 --2- 172.21.15.107:0/3082276648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c101790 0x7f710c101bb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 -- 172.21.15.107:0/3082276648 >> 172.21.15.107:0/3082276648 conn(0x7f710c0fcd20 msgr2=0x7f710c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 -- 172.21.15.107:0/3082276648 shutdown_connections 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.697+0000 7f7112a5d640 1 -- 172.21.15.107:0/3082276648 wait complete. 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 Processor -- start 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 -- start start 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c101790 0x7f710c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 0x7f710c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f710c10d400 con 0x7f710c103340 2024-04-24T23:45:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f710c10d280 con 0x7f710c102980 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7112a5d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f710c10d580 con 0x7f710c101790 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42862/0 (socket says 172.21.15.107:42862) 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 -- 172.21.15.107:0/274432320 learned_addr learned my addr 172.21.15.107:0/274432320 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f710bfff640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c101790 0x7f710c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f710b7fe640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 0x7f710c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 -- 172.21.15.107:0/274432320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c101790 msgr2=0x7f710c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c101790 0x7f710c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 -- 172.21.15.107:0/274432320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 msgr2=0x7f710c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 0x7f710c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f7110ffa640 1 -- 172.21.15.107:0/274432320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c19f720 con 0x7f710c103340 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.698+0000 7f710b7fe640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 0x7f710c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f7110ffa640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f710400eaa0 tx=0x7f710400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710400cdb0 con 0x7f710c103340 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7104005020 con 0x7f710c103340 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710c19fa10 con 0x7f710c103340 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71040116b0 con 0x7f710c103340 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.699+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710c19ff60 con 0x7f710c103340 2024-04-24T23:45:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.700+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710c108b50 con 0x7f710c103340 2024-04-24T23:45:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.701+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7104005560 con 0x7f710c103340 2024-04-24T23:45:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.701+0000 7f71097fa640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 0x7f70f40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.702+0000 7f710bfff640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 0x7f70f40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.702+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f71040cf600 con 0x7f710c103340 2024-04-24T23:45:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.702+0000 7f710bfff640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 0x7f70f40a50d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005ab0 tx=0x7f70fc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.706+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7104016030 con 0x7f710c103340 2024-04-24T23:45:17.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.880+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f710c062340 con 0x7f710c103340 2024-04-24T23:45:17.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.880+0000 7f71097fa640 1 -- 172.21.15.107:0/274432320 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f710409c190 con 0x7f710c103340 2024-04-24T23:45:17.880 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:45:17.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.882+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 msgr2=0x7f70f40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.882+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 0x7f70f40a50d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005ab0 tx=0x7f70fc000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.882+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 msgr2=0x7f710c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.882+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f710400eaa0 tx=0x7f710400ef60 comp rx=0 tx=0).stop 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 shutdown_connections 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f70f40a2c10 0x7f70f40a50d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c103340 0x7f710c19f0b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f710c102980 0x7f710c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 --2- 172.21.15.107:0/274432320 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f710c101790 0x7f710c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 >> 172.21.15.107:0/274432320 conn(0x7f710c0fcd20 msgr2=0x7f710c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 shutdown_connections 2024-04-24T23:45:17.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.883+0000 7f7112a5d640 1 -- 172.21.15.107:0/274432320 wait complete. 2024-04-24T23:45:17.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:45:17.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 -- 172.21.15.107:0/2523768239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0102230 msgr2=0x7fd8b0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 --2- 172.21.15.107:0/2523768239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0102230 0x7fd8b0108ac0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009ca0 tx=0x7fd8a802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 -- 172.21.15.107:0/2523768239 shutdown_connections 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 --2- 172.21.15.107:0/2523768239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0102230 0x7fd8b0108ac0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 --2- 172.21.15.107:0/2523768239 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8b0101870 0x7fd8b0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 --2- 172.21.15.107:0/2523768239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b0100680 0x7fd8b0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 -- 172.21.15.107:0/2523768239 >> 172.21.15.107:0/2523768239 conn(0x7fd8b00fbbf0 msgr2=0x7fd8b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 -- 172.21.15.107:0/2523768239 shutdown_connections 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.995+0000 7fd8b599e640 1 -- 172.21.15.107:0/2523768239 wait complete. 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 Processor -- start 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 -- start start 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8b0101870 0x7fd8b0196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b0102230 0x7fd8b019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:17.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010c300 con 0x7fd8b0100680 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010c180 con 0x7fd8b0102230 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8b599e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010c480 con 0x7fd8b0101870 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42882/0 (socket says 172.21.15.107:42882) 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 -- 172.21.15.107:0/3954708591 learned_addr learned my addr 172.21.15.107:0/3954708591 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 -- 172.21.15.107:0/3954708591 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8b0101870 msgr2=0x7fd8b0196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8b0101870 0x7fd8b0196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 -- 172.21.15.107:0/3954708591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b0102230 msgr2=0x7fd8b019dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b0102230 0x7fd8b019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.996+0000 7fd8aeffd640 1 -- 172.21.15.107:0/3954708591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b019e530 con 0x7fd8b0100680 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8aeffd640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c970 tx=0x7fd8a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0007c10 con 0x7fd8b0100680 2024-04-24T23:45:17.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a0007db0 con 0x7fd8b0100680 2024-04-24T23:45:18.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b019e820 con 0x7fd8b0100680 2024-04-24T23:45:18.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a000f940 con 0x7fd8b0100680 2024-04-24T23:45:18.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.997+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b019ed70 con 0x7fd8b0100680 2024-04-24T23:45:18.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.998+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b0107940 con 0x7fd8b0100680 2024-04-24T23:45:18.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.999+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd8a000fae0 con 0x7fd8b0100680 2024-04-24T23:45:18.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.999+0000 7fd8b499c640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 0x7fd89c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.999+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd8a00cf8c0 con 0x7fd8b0100680 2024-04-24T23:45:18.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:17.999+0000 7fd8a7fff640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 0x7fd89c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.000+0000 7fd8a7fff640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 0x7fd89c0a5050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0197e20 tx=0x7fd898005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:18.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.004+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8a009c4e0 con 0x7fd8b0100680 2024-04-24T23:45:18.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.179+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd8b002d170 con 0x7fd8b0100680 2024-04-24T23:45:18.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.179+0000 7fd8b499c640 1 -- 172.21.15.107:0/3954708591 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd8a009bde0 con 0x7fd8b0100680 2024-04-24T23:45:18.179 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:18.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.181+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 msgr2=0x7fd89c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.181+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 0x7fd89c0a5050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0197e20 tx=0x7fd898005bf0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.181+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 msgr2=0x7fd8b0196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.181+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c970 tx=0x7fd8a000ce30 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 shutdown_connections 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd89c0a2b90 0x7fd89c0a5050 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b0102230 0x7fd8b019dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd8b0101870 0x7fd8b0196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 --2- 172.21.15.107:0/3954708591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b0100680 0x7fd8b0196910 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 >> 172.21.15.107:0/3954708591 conn(0x7fd8b00fbbf0 msgr2=0x7fd8b0104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 shutdown_connections 2024-04-24T23:45:18.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.182+0000 7fd8b599e640 1 -- 172.21.15.107:0/3954708591 wait complete. 2024-04-24T23:45:18.193 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-24T23:45:18.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/4104550995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 msgr2=0x7f7b54109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/4104550995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b54109bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009900 tx=0x7f7b5002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/4104550995 shutdown_connections 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/4104550995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b54109bf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/4104550995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 0x7f7b54102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/4104550995 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/4104550995 >> 172.21.15.107:0/4104550995 conn(0x7f7b540fcd40 msgr2=0x7f7b540ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/4104550995 shutdown_connections 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/4104550995 wait complete. 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 Processor -- start 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- start start 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 0x7f7b54198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5410d430 con 0x7f7b54103360 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5410d2b0 con 0x7f7b541029a0 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.292+0000 7f7b5bb1d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5410d5b0 con 0x7f7b541017b0 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42910/0 (socket says 172.21.15.107:42910) 2024-04-24T23:45:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 -- 172.21.15.107:0/2095203921 learned_addr learned my addr 172.21.15.107:0/2095203921 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b590b8640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 0x7f7b54198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b598b9640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 -- 172.21.15.107:0/2095203921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 msgr2=0x7f7b54197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 -- 172.21.15.107:0/2095203921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 msgr2=0x7f7b54198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 0x7f7b54198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 -- 172.21.15.107:0/2095203921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5419f700 con 0x7f7b54103360 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b598b9640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5a0ba640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009230 tx=0x7f7b50004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5003f030 con 0x7f7b54103360 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b50003ce0 con 0x7f7b54103360 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5419f990 con 0x7f7b54103360 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50035070 con 0x7f7b54103360 2024-04-24T23:45:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.293+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5419fe80 con 0x7f7b54103360 2024-04-24T23:45:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.294+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b54108a70 con 0x7f7b54103360 2024-04-24T23:45:18.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.295+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7b50042030 con 0x7f7b54103360 2024-04-24T23:45:18.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.296+0000 7f7b4affd640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 0x7f7b400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.296+0000 7f7b598b9640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 0x7f7b400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.296+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7b500efe60 con 0x7f7b54103360 2024-04-24T23:45:18.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.296+0000 7f7b598b9640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 0x7f7b400a5050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7b54102800 tx=0x7f7b4c009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:18.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.299+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b500bc9f0 con 0x7f7b54103360 2024-04-24T23:45:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.522+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b5402d330 con 0x7f7b54103360 2024-04-24T23:45:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.523+0000 7f7b4affd640 1 -- 172.21.15.107:0/2095203921 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7f7b500bc2f0 con 0x7f7b54103360 2024-04-24T23:45:18.522 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:18.523 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:45:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 msgr2=0x7f7b400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 0x7f7b400a5050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7b54102800 tx=0x7f7b4c009520 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 msgr2=0x7f7b5419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009230 tx=0x7f7b50004360 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 shutdown_connections 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7b400a2b90 0x7f7b400a5050 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b54103360 0x7f7b5419f090 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b541029a0 0x7f7b54198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 --2- 172.21.15.107:0/2095203921 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7b541017b0 0x7f7b54197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 >> 172.21.15.107:0/2095203921 conn(0x7f7b540fcd40 msgr2=0x7f7b54105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 shutdown_connections 2024-04-24T23:45:18.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.526+0000 7f7b5bb1d640 1 -- 172.21.15.107:0/2095203921 wait complete. 2024-04-24T23:45:18.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1517468161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1251108641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:18 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/274432320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.541 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-24T23:45:18.541 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-24T23:45:18.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:45:18.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1517468161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1251108641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:18 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/274432320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1517468161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1251108641' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:18 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/274432320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- 172.21.15.107:0/3735379978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 msgr2=0x7f5d1c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- 172.21.15.107:0/3735379978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c1098b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009ca0 tx=0x7f5d1402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- 172.21.15.107:0/3735379978 shutdown_connections 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- 172.21.15.107:0/3735379978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c1098b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- 172.21.15.107:0/3735379978 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c104b30 0x7f5d1c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- 172.21.15.107:0/3735379978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c102200 0x7f5d1c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- 172.21.15.107:0/3735379978 >> 172.21.15.107:0/3735379978 conn(0x7f5d1c0fbbf0 msgr2=0x7f5d1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- 172.21.15.107:0/3735379978 shutdown_connections 2024-04-24T23:45:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- 172.21.15.107:0/3735379978 wait complete. 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 Processor -- start 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- start start 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c102200 0x7f5d1c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 0x7f5d1c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c10d420 con 0x7f5d1c107460 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c10d2a0 con 0x7f5d1c104b30 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.642+0000 7f5d24529640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c10d5a0 con 0x7f5d1c102200 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d22ac6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d22ac6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42922/0 (socket says 172.21.15.107:42922) 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 0x7f5d1c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d222c5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c102200 0x7f5d1c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d22ac6640 1 -- 172.21.15.107:0/1043478596 learned_addr learned my addr 172.21.15.107:0/1043478596 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 -- 172.21.15.107:0/1043478596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c102200 msgr2=0x7f5d1c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c102200 0x7f5d1c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 -- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 msgr2=0x7f5d1c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 -- 172.21.15.107:0/1043478596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c19f660 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d22ac6640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d21ac4640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 0x7f5d1c197f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9c0 tx=0x7f5d1800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800ccc0 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d18005020 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c19f950 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18004a00 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.643+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c19fe50 con 0x7f5d1c104b30 2024-04-24T23:45:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.644+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1c0fd6e0 con 0x7f5d1c104b30 2024-04-24T23:45:18.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.645+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5d180051d0 con 0x7f5d1c104b30 2024-04-24T23:45:18.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.646+0000 7f5d137fe640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 0x7f5d080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:18.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.646+0000 7f5d222c5640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 0x7f5d080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:18.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.646+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5d180cf3a0 con 0x7f5d1c104b30 2024-04-24T23:45:18.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.646+0000 7f5d222c5640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 0x7f5d080a5050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009bd0 tx=0x7f5d0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:18.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.651+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mon.2 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d1809bed0 con 0x7f5d1c104b30 2024-04-24T23:45:18.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.836+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d1c061b80 con 0x7f5d080a2b90 2024-04-24T23:45:18.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.846+0000 7f5d137fe640 1 -- 172.21.15.107:0/1043478596 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340833 (secure 0 0 0) 0x7f5d1c061b80 con 0x7f5d080a2b90 2024-04-24T23:45:18.850 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:18.852 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 msgr2=0x7f5d080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 0x7f5d080a5050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009bd0 tx=0x7f5d0c005f90 comp rx=0 tx=0).stop 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 msgr2=0x7f5d1c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 0x7f5d1c197f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800e9c0 tx=0x7f5d1800ee80 comp rx=0 tx=0).stop 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 shutdown_connections 2024-04-24T23:45:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5d080a2b90 0x7f5d080a5050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d1c107460 0x7f5d1c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d1c104b30 0x7f5d1c197f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 --2- 172.21.15.107:0/1043478596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d1c102200 0x7f5d1c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 >> 172.21.15.107:0/1043478596 conn(0x7f5d1c0fbbf0 msgr2=0x7f5d1c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 shutdown_connections 2024-04-24T23:45:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:18.850+0000 7f5d24529640 1 -- 172.21.15.107:0/1043478596 wait complete. 2024-04-24T23:45:18.863 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-24T23:45:18.013428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666127,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1331,"num_read_kb":794,"num_write":438,"num_write_kb":3096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1180848,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365964,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575860,"statfs":{"total":1151940427776,"available":1151565680640,"internally_reserved":0,"allocated":12894208,"data_stored":8206382,"data_compressed":432924,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":62,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003398"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190209+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:44:50.190209+0000","last_peered":"2024-04-24T23:44:50.190209+0000","last_clean":"2024-04-24T23:44:50.190209+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:44:50.190209+0000","last_undegraded":"2024-04-24T23:44:50.190209+0000","last_fullsized":"2024-04-24T23:44:50.190209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276613+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:44:50.276613+0000","last_peered":"2024-04-24T23:44:50.276613+0000","last_clean":"2024-04-24T23:44:50.276613+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:44:50.276613+0000","last_undegraded":"2024-04-24T23:44:50.276613+0000","last_fullsized":"2024-04-24T23:44:50.276613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276502+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:44:50.276502+0000","last_peered":"2024-04-24T23:44:50.276502+0000","last_clean":"2024-04-24T23:44:50.276502+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:44:50.276502+0000","last_undegraded":"2024-04-24T23:44:50.276502+0000","last_fullsized":"2024-04-24T23:44:50.276502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072084+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:44:50.072084+0000","last_peered":"2024-04-24T23:44:50.072084+0000","last_clean":"2024-04-24T23:44:50.072084+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:44:50.072084+0000","last_undegraded":"2024-04-24T23:44:50.072084+0000","last_fullsized":"2024-04-24T23:44:50.072084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073969+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:44:50.073969+0000","last_peered":"2024-04-24T23:44:50.073969+0000","last_clean":"2024-04-24T23:44:50.073969+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:44:50.073969+0000","last_undegraded":"2024-04-24T23:44:50.073969+0000","last_fullsized":"2024-04-24T23:44:50.073969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226383+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:44:50.226383+0000","last_peered":"2024-04-24T23:44:50.226383+0000","last_clean":"2024-04-24T23:44:50.226383+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:44:50.226383+0000","last_undegraded":"2024-04-24T23:44:50.226383+0000","last_fullsized":"2024-04-24T23:44:50.226383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227750+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:44:50.227750+0000","last_peered":"2024-04-24T23:44:50.227750+0000","last_clean":"2024-04-24T23:44:50.227750+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:44:50.227750+0000","last_undegraded":"2024-04-24T23:44:50.227750+0000","last_fullsized":"2024-04-24T23:44:50.227750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227133+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:44:50.227133+0000","last_peered":"2024-04-24T23:44:50.227133+0000","last_clean":"2024-04-24T23:44:50.227133+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:44:50.227133+0000","last_undegraded":"2024-04-24T23:44:50.227133+0000","last_fullsized":"2024-04-24T23:44:50.227133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224664+0000","last_change":"2024-04-24T23:43:49.227407+0000","last_active":"2024-04-24T23:44:50.224664+0000","last_peered":"2024-04-24T23:44:50.224664+0000","last_clean":"2024-04-24T23:44:50.224664+0000","last_became_active":"2024-04-24T23:43:49.227076+0000","last_became_peered":"2024-04-24T23:43:49.227076+0000","last_unstale":"2024-04-24T23:44:50.224664+0000","last_undegraded":"2024-04-24T23:44:50.224664+0000","last_fullsized":"2024-04-24T23:44:50.224664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:24.426521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227028+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:44:50.227028+0000","last_peered":"2024-04-24T23:44:50.227028+0000","last_clean":"2024-04-24T23:44:50.227028+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:44:50.227028+0000","last_undegraded":"2024-04-24T23:44:50.227028+0000","last_fullsized":"2024-04-24T23:44:50.227028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228179+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:44:50.228179+0000","last_peered":"2024-04-24T23:44:50.228179+0000","last_clean":"2024-04-24T23:44:50.228179+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:44:50.228179+0000","last_undegraded":"2024-04-24T23:44:50.228179+0000","last_fullsized":"2024-04-24T23:44:50.228179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071960+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:44:50.071960+0000","last_peered":"2024-04-24T23:44:50.071960+0000","last_clean":"2024-04-24T23:44:50.071960+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:44:50.071960+0000","last_undegraded":"2024-04-24T23:44:50.071960+0000","last_fullsized":"2024-04-24T23:44:50.071960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072824+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:44:50.072824+0000","last_peered":"2024-04-24T23:44:50.072824+0000","last_clean":"2024-04-24T23:44:50.072824+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:44:50.072824+0000","last_undegraded":"2024-04-24T23:44:50.072824+0000","last_fullsized":"2024-04-24T23:44:50.072824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276286+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:44:50.276286+0000","last_peered":"2024-04-24T23:44:50.276286+0000","last_clean":"2024-04-24T23:44:50.276286+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:44:50.276286+0000","last_undegraded":"2024-04-24T23:44:50.276286+0000","last_fullsized":"2024-04-24T23:44:50.276286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276668+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:44:50.276668+0000","last_peered":"2024-04-24T23:44:50.276668+0000","last_clean":"2024-04-24T23:44:50.276668+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:44:50.276668+0000","last_undegraded":"2024-04-24T23:44:50.276668+0000","last_fullsized":"2024-04-24T23:44:50.276668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073870+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:44:50.073870+0000","last_peered":"2024-04-24T23:44:50.073870+0000","last_clean":"2024-04-24T23:44:50.073870+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:44:50.073870+0000","last_undegraded":"2024-04-24T23:44:50.073870+0000","last_fullsized":"2024-04-24T23:44:50.073870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190155+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:44:50.190155+0000","last_peered":"2024-04-24T23:44:50.190155+0000","last_clean":"2024-04-24T23:44:50.190155+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:44:50.190155+0000","last_undegraded":"2024-04-24T23:44:50.190155+0000","last_fullsized":"2024-04-24T23:44:50.190155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224849+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:44:50.224849+0000","last_peered":"2024-04-24T23:44:50.224849+0000","last_clean":"2024-04-24T23:44:50.224849+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:44:50.224849+0000","last_undegraded":"2024-04-24T23:44:50.224849+0000","last_fullsized":"2024-04-24T23:44:50.224849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078718+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:44:50.078718+0000","last_peered":"2024-04-24T23:44:50.078718+0000","last_clean":"2024-04-24T23:44:50.078718+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:44:50.078718+0000","last_undegraded":"2024-04-24T23:44:50.078718+0000","last_fullsized":"2024-04-24T23:44:50.078718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.277581+0000","last_change":"2024-04-24T23:44:50.277581+0000","last_active":"2024-04-24T23:44:50.277581+0000","last_peered":"2024-04-24T23:44:50.277581+0000","last_clean":"2024-04-24T23:44:50.277581+0000","last_became_active":"2024-04-24T23:44:50.277201+0000","last_became_peered":"2024-04-24T23:44:50.277201+0000","last_unstale":"2024-04-24T23:44:50.277581+0000","last_undegraded":"2024-04-24T23:44:50.277581+0000","last_fullsized":"2024-04-24T23:44:50.277581+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:58.685328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:14.257865+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:45:14.257865+0000","last_peered":"2024-04-24T23:45:14.257865+0000","last_clean":"2024-04-24T23:45:14.257865+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:45:14.257865+0000","last_undegraded":"2024-04-24T23:45:14.257865+0000","last_fullsized":"2024-04-24T23:45:14.257865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227525+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:44:50.227525+0000","last_peered":"2024-04-24T23:44:50.227525+0000","last_clean":"2024-04-24T23:44:50.227525+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:44:50.227525+0000","last_undegraded":"2024-04-24T23:44:50.227525+0000","last_fullsized":"2024-04-24T23:44:50.227525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071427+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:44:50.071427+0000","last_peered":"2024-04-24T23:44:50.071427+0000","last_clean":"2024-04-24T23:44:50.071427+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:44:50.071427+0000","last_undegraded":"2024-04-24T23:44:50.071427+0000","last_fullsized":"2024-04-24T23:44:50.071427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226823+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:44:50.226823+0000","last_peered":"2024-04-24T23:44:50.226823+0000","last_clean":"2024-04-24T23:44:50.226823+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:44:50.226823+0000","last_undegraded":"2024-04-24T23:44:50.226823+0000","last_fullsized":"2024-04-24T23:44:50.226823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276593+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:44:50.276593+0000","last_peered":"2024-04-24T23:44:50.276593+0000","last_clean":"2024-04-24T23:44:50.276593+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:44:50.276593+0000","last_undegraded":"2024-04-24T23:44:50.276593+0000","last_fullsized":"2024-04-24T23:44:50.276593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071625+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:44:50.071625+0000","last_peered":"2024-04-24T23:44:50.071625+0000","last_clean":"2024-04-24T23:44:50.071625+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:44:50.071625+0000","last_undegraded":"2024-04-24T23:44:50.071625+0000","last_fullsized":"2024-04-24T23:44:50.071625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074618+0000","last_change":"2024-04-24T23:43:49.235090+0000","last_active":"2024-04-24T23:44:50.074618+0000","last_peered":"2024-04-24T23:44:50.074618+0000","last_clean":"2024-04-24T23:44:50.074618+0000","last_became_active":"2024-04-24T23:43:49.223773+0000","last_became_peered":"2024-04-24T23:43:49.223773+0000","last_unstale":"2024-04-24T23:44:50.074618+0000","last_undegraded":"2024-04-24T23:44:50.074618+0000","last_fullsized":"2024-04-24T23:44:50.074618+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:23.776176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074166+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:44:50.074166+0000","last_peered":"2024-04-24T23:44:50.074166+0000","last_clean":"2024-04-24T23:44:50.074166+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:44:50.074166+0000","last_undegraded":"2024-04-24T23:44:50.074166+0000","last_fullsized":"2024-04-24T23:44:50.074166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071760+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:44:50.071760+0000","last_peered":"2024-04-24T23:44:50.071760+0000","last_clean":"2024-04-24T23:44:50.071760+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:44:50.071760+0000","last_undegraded":"2024-04-24T23:44:50.071760+0000","last_fullsized":"2024-04-24T23:44:50.071760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073467+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:44:50.073467+0000","last_peered":"2024-04-24T23:44:50.073467+0000","last_clean":"2024-04-24T23:44:50.073467+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:44:50.073467+0000","last_undegraded":"2024-04-24T23:44:50.073467+0000","last_fullsized":"2024-04-24T23:44:50.073467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071740+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:44:50.071740+0000","last_peered":"2024-04-24T23:44:50.071740+0000","last_clean":"2024-04-24T23:44:50.071740+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:44:50.071740+0000","last_undegraded":"2024-04-24T23:44:50.071740+0000","last_fullsized":"2024-04-24T23:44:50.071740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078499+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:44:50.078499+0000","last_peered":"2024-04-24T23:44:50.078499+0000","last_clean":"2024-04-24T23:44:50.078499+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:44:50.078499+0000","last_undegraded":"2024-04-24T23:44:50.078499+0000","last_fullsized":"2024-04-24T23:44:50.078499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:16.262165+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:45:16.262165+0000","last_peered":"2024-04-24T23:45:16.262165+0000","last_clean":"2024-04-24T23:45:16.262165+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:45:16.262165+0000","last_undegraded":"2024-04-24T23:45:16.262165+0000","last_fullsized":"2024-04-24T23:45:16.262165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073965+0000","last_change":"2024-04-24T23:43:49.224709+0000","last_active":"2024-04-24T23:44:50.073965+0000","last_peered":"2024-04-24T23:44:50.073965+0000","last_clean":"2024-04-24T23:44:50.073965+0000","last_became_active":"2024-04-24T23:43:49.224345+0000","last_became_peered":"2024-04-24T23:43:49.224345+0000","last_unstale":"2024-04-24T23:44:50.073965+0000","last_undegraded":"2024-04-24T23:44:50.073965+0000","last_fullsized":"2024-04-24T23:44:50.073965+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:56.093532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078701+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:44:50.078701+0000","last_peered":"2024-04-24T23:44:50.078701+0000","last_clean":"2024-04-24T23:44:50.078701+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:44:50.078701+0000","last_undegraded":"2024-04-24T23:44:50.078701+0000","last_fullsized":"2024-04-24T23:44:50.078701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226287+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:44:50.226287+0000","last_peered":"2024-04-24T23:44:50.226287+0000","last_clean":"2024-04-24T23:44:50.226287+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:44:50.226287+0000","last_undegraded":"2024-04-24T23:44:50.226287+0000","last_fullsized":"2024-04-24T23:44:50.226287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071791+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:44:50.071791+0000","last_peered":"2024-04-24T23:44:50.071791+0000","last_clean":"2024-04-24T23:44:50.071791+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:44:50.071791+0000","last_undegraded":"2024-04-24T23:44:50.071791+0000","last_fullsized":"2024-04-24T23:44:50.071791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190050+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:44:50.190050+0000","last_peered":"2024-04-24T23:44:50.190050+0000","last_clean":"2024-04-24T23:44:50.190050+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:44:50.190050+0000","last_undegraded":"2024-04-24T23:44:50.190050+0000","last_fullsized":"2024-04-24T23:44:50.190050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072614+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:44:50.072614+0000","last_peered":"2024-04-24T23:44:50.072614+0000","last_clean":"2024-04-24T23:44:50.072614+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:44:50.072614+0000","last_undegraded":"2024-04-24T23:44:50.072614+0000","last_fullsized":"2024-04-24T23:44:50.072614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224223+0000","last_change":"2024-04-24T23:43:49.226164+0000","last_active":"2024-04-24T23:44:50.224223+0000","last_peered":"2024-04-24T23:44:50.224223+0000","last_clean":"2024-04-24T23:44:50.224223+0000","last_became_active":"2024-04-24T23:43:49.225806+0000","last_became_peered":"2024-04-24T23:43:49.225806+0000","last_unstale":"2024-04-24T23:44:50.224223+0000","last_undegraded":"2024-04-24T23:44:50.224223+0000","last_fullsized":"2024-04-24T23:44:50.224223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:56.068947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276432+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:44:50.276432+0000","last_peered":"2024-04-24T23:44:50.276432+0000","last_clean":"2024-04-24T23:44:50.276432+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:44:50.276432+0000","last_undegraded":"2024-04-24T23:44:50.276432+0000","last_fullsized":"2024-04-24T23:44:50.276432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071398+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:44:50.071398+0000","last_peered":"2024-04-24T23:44:50.071398+0000","last_clean":"2024-04-24T23:44:50.071398+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:44:50.071398+0000","last_undegraded":"2024-04-24T23:44:50.071398+0000","last_fullsized":"2024-04-24T23:44:50.071398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224569+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:44:50.224569+0000","last_peered":"2024-04-24T23:44:50.224569+0000","last_clean":"2024-04-24T23:44:50.224569+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:44:50.224569+0000","last_undegraded":"2024-04-24T23:44:50.224569+0000","last_fullsized":"2024-04-24T23:44:50.224569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227216+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:44:50.227216+0000","last_peered":"2024-04-24T23:44:50.227216+0000","last_clean":"2024-04-24T23:44:50.227216+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:44:50.227216+0000","last_undegraded":"2024-04-24T23:44:50.227216+0000","last_fullsized":"2024-04-24T23:44:50.227216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.303688+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:45:15.303688+0000","last_peered":"2024-04-24T23:45:15.303688+0000","last_clean":"2024-04-24T23:45:15.303688+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:45:15.303688+0000","last_undegraded":"2024-04-24T23:45:15.303688+0000","last_fullsized":"2024-04-24T23:45:15.303688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074660+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:44:50.074660+0000","last_peered":"2024-04-24T23:44:50.074660+0000","last_clean":"2024-04-24T23:44:50.074660+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:44:50.074660+0000","last_undegraded":"2024-04-24T23:44:50.074660+0000","last_fullsized":"2024-04-24T23:44:50.074660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224398+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:44:50.224398+0000","last_peered":"2024-04-24T23:44:50.224398+0000","last_clean":"2024-04-24T23:44:50.224398+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:44:50.224398+0000","last_undegraded":"2024-04-24T23:44:50.224398+0000","last_fullsized":"2024-04-24T23:44:50.224398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228258+0000","last_change":"2024-04-24T23:43:49.226781+0000","last_active":"2024-04-24T23:44:50.228258+0000","last_peered":"2024-04-24T23:44:50.228258+0000","last_clean":"2024-04-24T23:44:50.228258+0000","last_became_active":"2024-04-24T23:43:49.222089+0000","last_became_peered":"2024-04-24T23:43:49.222089+0000","last_unstale":"2024-04-24T23:44:50.228258+0000","last_undegraded":"2024-04-24T23:44:50.228258+0000","last_fullsized":"2024-04-24T23:44:50.228258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:57.224621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276509+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:44:50.276509+0000","last_peered":"2024-04-24T23:44:50.276509+0000","last_clean":"2024-04-24T23:44:50.276509+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:44:50.276509+0000","last_undegraded":"2024-04-24T23:44:50.276509+0000","last_fullsized":"2024-04-24T23:44:50.276509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071804+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:44:50.071804+0000","last_peered":"2024-04-24T23:44:50.071804+0000","last_clean":"2024-04-24T23:44:50.071804+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:44:50.071804+0000","last_undegraded":"2024-04-24T23:44:50.071804+0000","last_fullsized":"2024-04-24T23:44:50.071804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071732+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:44:50.071732+0000","last_peered":"2024-04-24T23:44:50.071732+0000","last_clean":"2024-04-24T23:44:50.071732+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:44:50.071732+0000","last_undegraded":"2024-04-24T23:44:50.071732+0000","last_fullsized":"2024-04-24T23:44:50.071732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074559+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:44:50.074559+0000","last_peered":"2024-04-24T23:44:50.074559+0000","last_clean":"2024-04-24T23:44:50.074559+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:44:50.074559+0000","last_undegraded":"2024-04-24T23:44:50.074559+0000","last_fullsized":"2024-04-24T23:44:50.074559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224451+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:44:50.224451+0000","last_peered":"2024-04-24T23:44:50.224451+0000","last_clean":"2024-04-24T23:44:50.224451+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:44:50.224451+0000","last_undegraded":"2024-04-24T23:44:50.224451+0000","last_fullsized":"2024-04-24T23:44:50.224451+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'41","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.260922+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:45:15.260922+0000","last_peered":"2024-04-24T23:45:15.260922+0000","last_clean":"2024-04-24T23:45:15.260922+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:45:15.260922+0000","last_undegraded":"2024-04-24T23:45:15.260922+0000","last_fullsized":"2024-04-24T23:45:15.260922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68263,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":189,"num_write":43,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074503+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:44:50.074503+0000","last_peered":"2024-04-24T23:44:50.074503+0000","last_clean":"2024-04-24T23:44:50.074503+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:44:50.074503+0000","last_undegraded":"2024-04-24T23:44:50.074503+0000","last_fullsized":"2024-04-24T23:44:50.074503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072028+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:44:50.072028+0000","last_peered":"2024-04-24T23:44:50.072028+0000","last_clean":"2024-04-24T23:44:50.072028+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:44:50.072028+0000","last_undegraded":"2024-04-24T23:44:50.072028+0000","last_fullsized":"2024-04-24T23:44:50.072028+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078708+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:44:50.078708+0000","last_peered":"2024-04-24T23:44:50.078708+0000","last_clean":"2024-04-24T23:44:50.078708+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:44:50.078708+0000","last_undegraded":"2024-04-24T23:44:50.078708+0000","last_fullsized":"2024-04-24T23:44:50.078708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072796+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:44:50.072796+0000","last_peered":"2024-04-24T23:44:50.072796+0000","last_clean":"2024-04-24T23:44:50.072796+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:44:50.072796+0000","last_undegraded":"2024-04-24T23:44:50.072796+0000","last_fullsized":"2024-04-24T23:44:50.072796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189700+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:44:50.189700+0000","last_peered":"2024-04-24T23:44:50.189700+0000","last_clean":"2024-04-24T23:44:50.189700+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:44:50.189700+0000","last_undegraded":"2024-04-24T23:44:50.189700+0000","last_fullsized":"2024-04-24T23:44:50.189700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071783+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:44:50.071783+0000","last_peered":"2024-04-24T23:44:50.071783+0000","last_clean":"2024-04-24T23:44:50.071783+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:44:50.071783+0000","last_undegraded":"2024-04-24T23:44:50.071783+0000","last_fullsized":"2024-04-24T23:44:50.071783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228754+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:44:50.228754+0000","last_peered":"2024-04-24T23:44:50.228754+0000","last_clean":"2024-04-24T23:44:50.228754+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:44:50.228754+0000","last_undegraded":"2024-04-24T23:44:50.228754+0000","last_fullsized":"2024-04-24T23:44:50.228754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224947+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:44:50.224947+0000","last_peered":"2024-04-24T23:44:50.224947+0000","last_clean":"2024-04-24T23:44:50.224947+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:44:50.224947+0000","last_undegraded":"2024-04-24T23:44:50.224947+0000","last_fullsized":"2024-04-24T23:44:50.224947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.256361+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:45:12.256361+0000","last_peered":"2024-04-24T23:45:12.256361+0000","last_clean":"2024-04-24T23:45:12.256361+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:45:12.256361+0000","last_undegraded":"2024-04-24T23:45:12.256361+0000","last_fullsized":"2024-04-24T23:45:12.256361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228739+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:44:50.228739+0000","last_peered":"2024-04-24T23:44:50.228739+0000","last_clean":"2024-04-24T23:44:50.228739+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:44:50.228739+0000","last_undegraded":"2024-04-24T23:44:50.228739+0000","last_fullsized":"2024-04-24T23:44:50.228739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073186+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:44:50.073186+0000","last_peered":"2024-04-24T23:44:50.073186+0000","last_clean":"2024-04-24T23:44:50.073186+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:44:50.073186+0000","last_undegraded":"2024-04-24T23:44:50.073186+0000","last_fullsized":"2024-04-24T23:44:50.073186+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073794+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:44:50.073794+0000","last_peered":"2024-04-24T23:44:50.073794+0000","last_clean":"2024-04-24T23:44:50.073794+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:44:50.073794+0000","last_undegraded":"2024-04-24T23:44:50.073794+0000","last_fullsized":"2024-04-24T23:44:50.073794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071742+0000","last_change":"2024-04-24T23:43:49.223137+0000","last_active":"2024-04-24T23:44:50.071742+0000","last_peered":"2024-04-24T23:44:50.071742+0000","last_clean":"2024-04-24T23:44:50.071742+0000","last_became_active":"2024-04-24T23:43:49.222775+0000","last_became_peered":"2024-04-24T23:43:49.222775+0000","last_unstale":"2024-04-24T23:44:50.071742+0000","last_undegraded":"2024-04-24T23:44:50.071742+0000","last_fullsized":"2024-04-24T23:44:50.071742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:22.910320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071553+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:44:50.071553+0000","last_peered":"2024-04-24T23:44:50.071553+0000","last_clean":"2024-04-24T23:44:50.071553+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:44:50.071553+0000","last_undegraded":"2024-04-24T23:44:50.071553+0000","last_fullsized":"2024-04-24T23:44:50.071553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074870+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:44:50.074870+0000","last_peered":"2024-04-24T23:44:50.074870+0000","last_clean":"2024-04-24T23:44:50.074870+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:44:50.074870+0000","last_undegraded":"2024-04-24T23:44:50.074870+0000","last_fullsized":"2024-04-24T23:44:50.074870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190122+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:44:50.190122+0000","last_peered":"2024-04-24T23:44:50.190122+0000","last_clean":"2024-04-24T23:44:50.190122+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:44:50.190122+0000","last_undegraded":"2024-04-24T23:44:50.190122+0000","last_fullsized":"2024-04-24T23:44:50.190122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073941+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:44:50.073941+0000","last_peered":"2024-04-24T23:44:50.073941+0000","last_clean":"2024-04-24T23:44:50.073941+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:44:50.073941+0000","last_undegraded":"2024-04-24T23:44:50.073941+0000","last_fullsized":"2024-04-24T23:44:50.073941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224709+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:44:50.224709+0000","last_peered":"2024-04-24T23:44:50.224709+0000","last_clean":"2024-04-24T23:44:50.224709+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:44:50.224709+0000","last_undegraded":"2024-04-24T23:44:50.224709+0000","last_fullsized":"2024-04-24T23:44:50.224709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226822+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:44:50.226822+0000","last_peered":"2024-04-24T23:44:50.226822+0000","last_clean":"2024-04-24T23:44:50.226822+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:44:50.226822+0000","last_undegraded":"2024-04-24T23:44:50.226822+0000","last_fullsized":"2024-04-24T23:44:50.226822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072855+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:44:50.072855+0000","last_peered":"2024-04-24T23:44:50.072855+0000","last_clean":"2024-04-24T23:44:50.072855+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:44:50.072855+0000","last_undegraded":"2024-04-24T23:44:50.072855+0000","last_fullsized":"2024-04-24T23:44:50.072855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.257199+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:45:13.257199+0000","last_peered":"2024-04-24T23:45:13.257199+0000","last_clean":"2024-04-24T23:45:13.257199+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:45:13.257199+0000","last_undegraded":"2024-04-24T23:45:13.257199+0000","last_fullsized":"2024-04-24T23:45:13.257199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226654+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:44:50.226654+0000","last_peered":"2024-04-24T23:44:50.226654+0000","last_clean":"2024-04-24T23:44:50.226654+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:44:50.226654+0000","last_undegraded":"2024-04-24T23:44:50.226654+0000","last_fullsized":"2024-04-24T23:44:50.226654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073330+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:44:50.073330+0000","last_peered":"2024-04-24T23:44:50.073330+0000","last_clean":"2024-04-24T23:44:50.073330+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:44:50.073330+0000","last_undegraded":"2024-04-24T23:44:50.073330+0000","last_fullsized":"2024-04-24T23:44:50.073330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071894+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:44:50.071894+0000","last_peered":"2024-04-24T23:44:50.071894+0000","last_clean":"2024-04-24T23:44:50.071894+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:44:50.071894+0000","last_undegraded":"2024-04-24T23:44:50.071894+0000","last_fullsized":"2024-04-24T23:44:50.071894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.255804+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:45:12.255804+0000","last_peered":"2024-04-24T23:45:12.255804+0000","last_clean":"2024-04-24T23:45:12.255804+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:45:12.255804+0000","last_undegraded":"2024-04-24T23:45:12.255804+0000","last_fullsized":"2024-04-24T23:45:12.255804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228121+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:44:50.228121+0000","last_peered":"2024-04-24T23:44:50.228121+0000","last_clean":"2024-04-24T23:44:50.228121+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:44:50.228121+0000","last_undegraded":"2024-04-24T23:44:50.228121+0000","last_fullsized":"2024-04-24T23:44:50.228121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227286+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:44:50.227286+0000","last_peered":"2024-04-24T23:44:50.227286+0000","last_clean":"2024-04-24T23:44:50.227286+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:44:50.227286+0000","last_undegraded":"2024-04-24T23:44:50.227286+0000","last_fullsized":"2024-04-24T23:44:50.227286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071805+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:44:50.071805+0000","last_peered":"2024-04-24T23:44:50.071805+0000","last_clean":"2024-04-24T23:44:50.071805+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:44:50.071805+0000","last_undegraded":"2024-04-24T23:44:50.071805+0000","last_fullsized":"2024-04-24T23:44:50.071805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228428+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:44:50.228428+0000","last_peered":"2024-04-24T23:44:50.228428+0000","last_clean":"2024-04-24T23:44:50.228428+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:44:50.228428+0000","last_undegraded":"2024-04-24T23:44:50.228428+0000","last_fullsized":"2024-04-24T23:44:50.228428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072241+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:44:50.072241+0000","last_peered":"2024-04-24T23:44:50.072241+0000","last_clean":"2024-04-24T23:44:50.072241+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:44:50.072241+0000","last_undegraded":"2024-04-24T23:44:50.072241+0000","last_fullsized":"2024-04-24T23:44:50.072241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072552+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:44:50.072552+0000","last_peered":"2024-04-24T23:44:50.072552+0000","last_clean":"2024-04-24T23:44:50.072552+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:44:50.072552+0000","last_undegraded":"2024-04-24T23:44:50.072552+0000","last_fullsized":"2024-04-24T23:44:50.072552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227689+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:44:50.227689+0000","last_peered":"2024-04-24T23:44:50.227689+0000","last_clean":"2024-04-24T23:44:50.227689+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:44:50.227689+0000","last_undegraded":"2024-04-24T23:44:50.227689+0000","last_fullsized":"2024-04-24T23:44:50.227689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189830+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:44:50.189830+0000","last_peered":"2024-04-24T23:44:50.189830+0000","last_clean":"2024-04-24T23:44:50.189830+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:44:50.189830+0000","last_undegraded":"2024-04-24T23:44:50.189830+0000","last_fullsized":"2024-04-24T23:44:50.189830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189864+0000","last_change":"2024-04-24T23:43:49.235755+0000","last_active":"2024-04-24T23:44:50.189864+0000","last_peered":"2024-04-24T23:44:50.189864+0000","last_clean":"2024-04-24T23:44:50.189864+0000","last_became_active":"2024-04-24T23:43:49.230488+0000","last_became_peered":"2024-04-24T23:43:49.230488+0000","last_unstale":"2024-04-24T23:44:50.189864+0000","last_undegraded":"2024-04-24T23:44:50.189864+0000","last_fullsized":"2024-04-24T23:44:50.189864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:12.034199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073003+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:44:50.073003+0000","last_peered":"2024-04-24T23:44:50.073003+0000","last_clean":"2024-04-24T23:44:50.073003+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:44:50.073003+0000","last_undegraded":"2024-04-24T23:44:50.073003+0000","last_fullsized":"2024-04-24T23:44:50.073003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227292+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:44:50.227292+0000","last_peered":"2024-04-24T23:44:50.227292+0000","last_clean":"2024-04-24T23:44:50.227292+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:44:50.227292+0000","last_undegraded":"2024-04-24T23:44:50.227292+0000","last_fullsized":"2024-04-24T23:44:50.227292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:01.190467+0000","last_change":"2024-04-24T23:43:49.236439+0000","last_active":"2024-04-24T23:45:01.190467+0000","last_peered":"2024-04-24T23:45:01.190467+0000","last_clean":"2024-04-24T23:45:01.190467+0000","last_became_active":"2024-04-24T23:43:49.227756+0000","last_became_peered":"2024-04-24T23:43:49.227756+0000","last_unstale":"2024-04-24T23:45:01.190467+0000","last_undegraded":"2024-04-24T23:45:01.190467+0000","last_fullsized":"2024-04-24T23:45:01.190467+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:34.790165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227799+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:44:50.227799+0000","last_peered":"2024-04-24T23:44:50.227799+0000","last_clean":"2024-04-24T23:44:50.227799+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:44:50.227799+0000","last_undegraded":"2024-04-24T23:44:50.227799+0000","last_fullsized":"2024-04-24T23:44:50.227799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078047+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:44:50.078047+0000","last_peered":"2024-04-24T23:44:50.078047+0000","last_clean":"2024-04-24T23:44:50.078047+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:44:50.078047+0000","last_undegraded":"2024-04-24T23:44:50.078047+0000","last_fullsized":"2024-04-24T23:44:50.078047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071411+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:44:50.071411+0000","last_peered":"2024-04-24T23:44:50.071411+0000","last_clean":"2024-04-24T23:44:50.071411+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:44:50.071411+0000","last_undegraded":"2024-04-24T23:44:50.071411+0000","last_fullsized":"2024-04-24T23:44:50.071411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276642+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:44:50.276642+0000","last_peered":"2024-04-24T23:44:50.276642+0000","last_clean":"2024-04-24T23:44:50.276642+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:44:50.276642+0000","last_undegraded":"2024-04-24T23:44:50.276642+0000","last_fullsized":"2024-04-24T23:44:50.276642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228660+0000","last_change":"2024-04-24T23:44:50.228080+0000","last_active":"2024-04-24T23:44:50.228660+0000","last_peered":"2024-04-24T23:44:50.228660+0000","last_clean":"2024-04-24T23:44:50.228660+0000","last_became_active":"2024-04-24T23:44:50.227599+0000","last_became_peered":"2024-04-24T23:44:50.227599+0000","last_unstale":"2024-04-24T23:44:50.228660+0000","last_undegraded":"2024-04-24T23:44:50.228660+0000","last_fullsized":"2024-04-24T23:44:50.228660+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:49.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074277+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:44:50.074277+0000","last_peered":"2024-04-24T23:44:50.074277+0000","last_clean":"2024-04-24T23:44:50.074277+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:44:50.074277+0000","last_undegraded":"2024-04-24T23:44:50.074277+0000","last_fullsized":"2024-04-24T23:44:50.074277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073609+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:44:50.073609+0000","last_peered":"2024-04-24T23:44:50.073609+0000","last_clean":"2024-04-24T23:44:50.073609+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:44:50.073609+0000","last_undegraded":"2024-04-24T23:44:50.073609+0000","last_fullsized":"2024-04-24T23:44:50.073609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.229292+0000","last_change":"2024-04-24T23:44:50.229292+0000","last_active":"2024-04-24T23:44:50.229292+0000","last_peered":"2024-04-24T23:44:50.229292+0000","last_clean":"2024-04-24T23:44:50.229292+0000","last_became_active":"2024-04-24T23:44:50.228528+0000","last_became_peered":"2024-04-24T23:44:50.228528+0000","last_unstale":"2024-04-24T23:44:50.229292+0000","last_undegraded":"2024-04-24T23:44:50.229292+0000","last_fullsized":"2024-04-24T23:44:50.229292+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:53.700501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227345+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:44:50.227345+0000","last_peered":"2024-04-24T23:44:50.227345+0000","last_clean":"2024-04-24T23:44:50.227345+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:44:50.227345+0000","last_undegraded":"2024-04-24T23:44:50.227345+0000","last_fullsized":"2024-04-24T23:44:50.227345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189662+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:44:50.189662+0000","last_peered":"2024-04-24T23:44:50.189662+0000","last_clean":"2024-04-24T23:44:50.189662+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:44:50.189662+0000","last_undegraded":"2024-04-24T23:44:50.189662+0000","last_fullsized":"2024-04-24T23:44:50.189662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071954+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:44:50.071954+0000","last_peered":"2024-04-24T23:44:50.071954+0000","last_clean":"2024-04-24T23:44:50.071954+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:44:50.071954+0000","last_undegraded":"2024-04-24T23:44:50.071954+0000","last_fullsized":"2024-04-24T23:44:50.071954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.077868+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:44:50.077868+0000","last_peered":"2024-04-24T23:44:50.077868+0000","last_clean":"2024-04-24T23:44:50.077868+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:44:50.077868+0000","last_undegraded":"2024-04-24T23:44:50.077868+0000","last_fullsized":"2024-04-24T23:44:50.077868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227346+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:44:50.227346+0000","last_peered":"2024-04-24T23:44:50.227346+0000","last_clean":"2024-04-24T23:44:50.227346+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:44:50.227346+0000","last_undegraded":"2024-04-24T23:44:50.227346+0000","last_fullsized":"2024-04-24T23:44:50.227346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072175+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:44:50.072175+0000","last_peered":"2024-04-24T23:44:50.072175+0000","last_clean":"2024-04-24T23:44:50.072175+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:44:50.072175+0000","last_undegraded":"2024-04-24T23:44:50.072175+0000","last_fullsized":"2024-04-24T23:44:50.072175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074163+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:44:50.074163+0000","last_peered":"2024-04-24T23:44:50.074163+0000","last_clean":"2024-04-24T23:44:50.074163+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:44:50.074163+0000","last_undegraded":"2024-04-24T23:44:50.074163+0000","last_fullsized":"2024-04-24T23:44:50.074163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228420+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:44:50.228420+0000","last_peered":"2024-04-24T23:44:50.228420+0000","last_clean":"2024-04-24T23:44:50.228420+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:44:50.228420+0000","last_undegraded":"2024-04-24T23:44:50.228420+0000","last_fullsized":"2024-04-24T23:44:50.228420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.230519+0000","last_change":"2024-04-24T23:44:50.230008+0000","last_active":"2024-04-24T23:44:50.230519+0000","last_peered":"2024-04-24T23:44:50.230519+0000","last_clean":"2024-04-24T23:44:50.230519+0000","last_became_active":"2024-04-24T23:44:50.229452+0000","last_became_peered":"2024-04-24T23:44:50.229452+0000","last_unstale":"2024-04-24T23:44:50.230519+0000","last_undegraded":"2024-04-24T23:44:50.230519+0000","last_fullsized":"2024-04-24T23:44:50.230519+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:13.691069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.300487+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:45:13.300487+0000","last_peered":"2024-04-24T23:45:13.300487+0000","last_clean":"2024-04-24T23:45:13.300487+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:45:13.300487+0000","last_undegraded":"2024-04-24T23:45:13.300487+0000","last_fullsized":"2024-04-24T23:45:13.300487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1478,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228299+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:44:50.228299+0000","last_peered":"2024-04-24T23:44:50.228299+0000","last_clean":"2024-04-24T23:44:50.228299+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:44:50.228299+0000","last_undegraded":"2024-04-24T23:44:50.228299+0000","last_fullsized":"2024-04-24T23:44:50.228299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276473+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:44:50.276473+0000","last_peered":"2024-04-24T23:44:50.276473+0000","last_clean":"2024-04-24T23:44:50.276473+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:44:50.276473+0000","last_undegraded":"2024-04-24T23:44:50.276473+0000","last_fullsized":"2024-04-24T23:44:50.276473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071951+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:44:50.071951+0000","last_peered":"2024-04-24T23:44:50.071951+0000","last_clean":"2024-04-24T23:44:50.071951+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:44:50.071951+0000","last_undegraded":"2024-04-24T23:44:50.071951+0000","last_fullsized":"2024-04-24T23:44:50.071951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228018+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:44:50.228018+0000","last_peered":"2024-04-24T23:44:50.228018+0000","last_clean":"2024-04-24T23:44:50.228018+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:44:50.228018+0000","last_undegraded":"2024-04-24T23:44:50.228018+0000","last_fullsized":"2024-04-24T23:44:50.228018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071162+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:44:50.071162+0000","last_peered":"2024-04-24T23:44:50.071162+0000","last_clean":"2024-04-24T23:44:50.071162+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:44:50.071162+0000","last_undegraded":"2024-04-24T23:44:50.071162+0000","last_fullsized":"2024-04-24T23:44:50.071162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073134+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:44:50.073134+0000","last_peered":"2024-04-24T23:44:50.073134+0000","last_clean":"2024-04-24T23:44:50.073134+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:44:50.073134+0000","last_undegraded":"2024-04-24T23:44:50.073134+0000","last_fullsized":"2024-04-24T23:44:50.073134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228522+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:44:50.228522+0000","last_peered":"2024-04-24T23:44:50.228522+0000","last_clean":"2024-04-24T23:44:50.228522+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:44:50.228522+0000","last_undegraded":"2024-04-24T23:44:50.228522+0000","last_fullsized":"2024-04-24T23:44:50.228522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189686+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:44:50.189686+0000","last_peered":"2024-04-24T23:44:50.189686+0000","last_clean":"2024-04-24T23:44:50.189686+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:44:50.189686+0000","last_undegraded":"2024-04-24T23:44:50.189686+0000","last_fullsized":"2024-04-24T23:44:50.189686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078449+0000","last_change":"2024-04-24T23:43:49.223272+0000","last_active":"2024-04-24T23:44:50.078449+0000","last_peered":"2024-04-24T23:44:50.078449+0000","last_clean":"2024-04-24T23:44:50.078449+0000","last_became_active":"2024-04-24T23:43:49.222921+0000","last_became_peered":"2024-04-24T23:43:49.222921+0000","last_unstale":"2024-04-24T23:44:50.078449+0000","last_undegraded":"2024-04-24T23:44:50.078449+0000","last_fullsized":"2024-04-24T23:44:50.078449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:10.443359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078022+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:44:50.078022+0000","last_peered":"2024-04-24T23:44:50.078022+0000","last_clean":"2024-04-24T23:44:50.078022+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:44:50.078022+0000","last_undegraded":"2024-04-24T23:44:50.078022+0000","last_fullsized":"2024-04-24T23:44:50.078022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276474+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:44:50.276474+0000","last_peered":"2024-04-24T23:44:50.276474+0000","last_clean":"2024-04-24T23:44:50.276474+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:44:50.276474+0000","last_undegraded":"2024-04-24T23:44:50.276474+0000","last_fullsized":"2024-04-24T23:44:50.276474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072969+0000","last_change":"2024-04-24T23:43:49.230897+0000","last_active":"2024-04-24T23:44:50.072969+0000","last_peered":"2024-04-24T23:44:50.072969+0000","last_clean":"2024-04-24T23:44:50.072969+0000","last_became_active":"2024-04-24T23:43:49.230514+0000","last_became_peered":"2024-04-24T23:43:49.230514+0000","last_unstale":"2024-04-24T23:44:50.072969+0000","last_undegraded":"2024-04-24T23:44:50.072969+0000","last_fullsized":"2024-04-24T23:44:50.072969+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:24.826152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190172+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:44:50.190172+0000","last_peered":"2024-04-24T23:44:50.190172+0000","last_clean":"2024-04-24T23:44:50.190172+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:44:50.190172+0000","last_undegraded":"2024-04-24T23:44:50.190172+0000","last_fullsized":"2024-04-24T23:44:50.190172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.079053+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:44:50.079053+0000","last_peered":"2024-04-24T23:44:50.079053+0000","last_clean":"2024-04-24T23:44:50.079053+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:44:50.079053+0000","last_undegraded":"2024-04-24T23:44:50.079053+0000","last_fullsized":"2024-04-24T23:44:50.079053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227836+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:44:50.227836+0000","last_peered":"2024-04-24T23:44:50.227836+0000","last_clean":"2024-04-24T23:44:50.227836+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:44:50.227836+0000","last_undegraded":"2024-04-24T23:44:50.227836+0000","last_fullsized":"2024-04-24T23:44:50.227836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071656+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:44:50.071656+0000","last_peered":"2024-04-24T23:44:50.071656+0000","last_clean":"2024-04-24T23:44:50.071656+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:44:50.071656+0000","last_undegraded":"2024-04-24T23:44:50.071656+0000","last_fullsized":"2024-04-24T23:44:50.071656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227382+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:44:50.227382+0000","last_peered":"2024-04-24T23:44:50.227382+0000","last_clean":"2024-04-24T23:44:50.227382+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:44:50.227382+0000","last_undegraded":"2024-04-24T23:44:50.227382+0000","last_fullsized":"2024-04-24T23:44:50.227382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.075088+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:44:50.075088+0000","last_peered":"2024-04-24T23:44:50.075088+0000","last_clean":"2024-04-24T23:44:50.075088+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:44:50.075088+0000","last_undegraded":"2024-04-24T23:44:50.075088+0000","last_fullsized":"2024-04-24T23:44:50.075088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073092+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:44:50.073092+0000","last_peered":"2024-04-24T23:44:50.073092+0000","last_clean":"2024-04-24T23:44:50.073092+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:44:50.073092+0000","last_undegraded":"2024-04-24T23:44:50.073092+0000","last_fullsized":"2024-04-24T23:44:50.073092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189908+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:44:50.189908+0000","last_peered":"2024-04-24T23:44:50.189908+0000","last_clean":"2024-04-24T23:44:50.189908+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:44:50.189908+0000","last_undegraded":"2024-04-24T23:44:50.189908+0000","last_fullsized":"2024-04-24T23:44:50.189908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75525,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1205,"num_read_kb":685,"num_write":255,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":112,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":221202,"data_compressed":4866,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520817,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1176008,"data_compressed":64433,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":524423,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":520593,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":1117343,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515831,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":585124,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517595,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":580279,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":515539,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520437,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1112393,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":70015,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65170,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70531,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:45:18.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:45:19.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.003+0000 7fbf2de95640 1 -- 172.21.15.107:0/731431230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 msgr2=0x7fbf28100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.003+0000 7fbf2de95640 1 --2- 172.21.15.107:0/731431230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28100aa0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009900 tx=0x7fbf1802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- 172.21.15.107:0/731431230 shutdown_connections 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- 172.21.15.107:0/731431230 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28102230 0x7fbf28108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- 172.21.15.107:0/731431230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28101870 0x7fbf28101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- 172.21.15.107:0/731431230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28100aa0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- 172.21.15.107:0/731431230 >> 172.21.15.107:0/731431230 conn(0x7fbf280fbbf0 msgr2=0x7fbf280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- 172.21.15.107:0/731431230 shutdown_connections 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- 172.21.15.107:0/731431230 wait complete. 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 Processor -- start 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- start start 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28101870 0x7fbf2819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28102230 0x7fbf2819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810c2d0 con 0x7fbf28100680 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810c150 con 0x7fbf28102230 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.004+0000 7fbf2de95640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810c450 con 0x7fbf28101870 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42944/0 (socket says 172.21.15.107:42944) 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 -- 172.21.15.107:0/1495612430 learned_addr learned my addr 172.21.15.107:0/1495612430 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf27fff640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28102230 0x7fbf2819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf1effd640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28101870 0x7fbf2819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 -- 172.21.15.107:0/1495612430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28101870 msgr2=0x7fbf2819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28101870 0x7fbf2819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 -- 172.21.15.107:0/1495612430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28102230 msgr2=0x7fbf2819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28102230 0x7fbf2819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 -- 172.21.15.107:0/1495612430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2819f6c0 con 0x7fbf28100680 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf277fe640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbf18031610 tx=0x7fbf180043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1803f030 con 0x7fbf28100680 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2819f8f0 con 0x7fbf28100680 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf18003ce0 con 0x7fbf28100680 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf18035070 con 0x7fbf28100680 2024-04-24T23:45:19.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.005+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2819fe30 con 0x7fbf28100680 2024-04-24T23:45:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.006+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf281079a0 con 0x7fbf28100680 2024-04-24T23:45:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.007+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbf18042030 con 0x7fbf28100680 2024-04-24T23:45:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.008+0000 7fbf257fa640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 0x7fbf140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.008+0000 7fbf1effd640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 0x7fbf140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.008+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbf180ef9b0 con 0x7fbf28100680 2024-04-24T23:45:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.009+0000 7fbf1effd640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 0x7fbf140a5050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005fd0 tx=0x7fbf10005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.013+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf180bc540 con 0x7fbf28100680 2024-04-24T23:45:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.193+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf28061b80 con 0x7fbf140a2b90 2024-04-24T23:45:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.203+0000 7fbf257fa640 1 -- 172.21.15.107:0/1495612430 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340833 (secure 0 0 0) 0x7fbf28061b80 con 0x7fbf140a2b90 2024-04-24T23:45:19.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:19.208 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:45:19.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 msgr2=0x7fbf140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 0x7fbf140a5050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005fd0 tx=0x7fbf10005850 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 msgr2=0x7fbf28069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbf18031610 tx=0x7fbf180043b0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 shutdown_connections 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbf140a2b90 0x7fbf140a5050 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf28102230 0x7fbf2819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbf28101870 0x7fbf2819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 --2- 172.21.15.107:0/1495612430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf28100680 0x7fbf28069440 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 >> 172.21.15.107:0/1495612430 conn(0x7fbf280fbbf0 msgr2=0x7fbf28104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 shutdown_connections 2024-04-24T23:45:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.206+0000 7fbf2de95640 1 -- 172.21.15.107:0/1495612430 wait complete. 2024-04-24T23:45:19.218 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-24T23:45:18.013428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666127,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1331,"num_read_kb":794,"num_write":438,"num_write_kb":3096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1180848,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365964,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575860,"statfs":{"total":1151940427776,"available":1151565680640,"internally_reserved":0,"allocated":12894208,"data_stored":8206382,"data_compressed":432924,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":62,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003398"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190209+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:44:50.190209+0000","last_peered":"2024-04-24T23:44:50.190209+0000","last_clean":"2024-04-24T23:44:50.190209+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:44:50.190209+0000","last_undegraded":"2024-04-24T23:44:50.190209+0000","last_fullsized":"2024-04-24T23:44:50.190209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276613+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:44:50.276613+0000","last_peered":"2024-04-24T23:44:50.276613+0000","last_clean":"2024-04-24T23:44:50.276613+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:44:50.276613+0000","last_undegraded":"2024-04-24T23:44:50.276613+0000","last_fullsized":"2024-04-24T23:44:50.276613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276502+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:44:50.276502+0000","last_peered":"2024-04-24T23:44:50.276502+0000","last_clean":"2024-04-24T23:44:50.276502+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:44:50.276502+0000","last_undegraded":"2024-04-24T23:44:50.276502+0000","last_fullsized":"2024-04-24T23:44:50.276502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072084+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:44:50.072084+0000","last_peered":"2024-04-24T23:44:50.072084+0000","last_clean":"2024-04-24T23:44:50.072084+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:44:50.072084+0000","last_undegraded":"2024-04-24T23:44:50.072084+0000","last_fullsized":"2024-04-24T23:44:50.072084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073969+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:44:50.073969+0000","last_peered":"2024-04-24T23:44:50.073969+0000","last_clean":"2024-04-24T23:44:50.073969+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:44:50.073969+0000","last_undegraded":"2024-04-24T23:44:50.073969+0000","last_fullsized":"2024-04-24T23:44:50.073969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226383+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:44:50.226383+0000","last_peered":"2024-04-24T23:44:50.226383+0000","last_clean":"2024-04-24T23:44:50.226383+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:44:50.226383+0000","last_undegraded":"2024-04-24T23:44:50.226383+0000","last_fullsized":"2024-04-24T23:44:50.226383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227750+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:44:50.227750+0000","last_peered":"2024-04-24T23:44:50.227750+0000","last_clean":"2024-04-24T23:44:50.227750+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:44:50.227750+0000","last_undegraded":"2024-04-24T23:44:50.227750+0000","last_fullsized":"2024-04-24T23:44:50.227750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227133+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:44:50.227133+0000","last_peered":"2024-04-24T23:44:50.227133+0000","last_clean":"2024-04-24T23:44:50.227133+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:44:50.227133+0000","last_undegraded":"2024-04-24T23:44:50.227133+0000","last_fullsized":"2024-04-24T23:44:50.227133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224664+0000","last_change":"2024-04-24T23:43:49.227407+0000","last_active":"2024-04-24T23:44:50.224664+0000","last_peered":"2024-04-24T23:44:50.224664+0000","last_clean":"2024-04-24T23:44:50.224664+0000","last_became_active":"2024-04-24T23:43:49.227076+0000","last_became_peered":"2024-04-24T23:43:49.227076+0000","last_unstale":"2024-04-24T23:44:50.224664+0000","last_undegraded":"2024-04-24T23:44:50.224664+0000","last_fullsized":"2024-04-24T23:44:50.224664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:24.426521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227028+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:44:50.227028+0000","last_peered":"2024-04-24T23:44:50.227028+0000","last_clean":"2024-04-24T23:44:50.227028+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:44:50.227028+0000","last_undegraded":"2024-04-24T23:44:50.227028+0000","last_fullsized":"2024-04-24T23:44:50.227028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228179+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:44:50.228179+0000","last_peered":"2024-04-24T23:44:50.228179+0000","last_clean":"2024-04-24T23:44:50.228179+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:44:50.228179+0000","last_undegraded":"2024-04-24T23:44:50.228179+0000","last_fullsized":"2024-04-24T23:44:50.228179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071960+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:44:50.071960+0000","last_peered":"2024-04-24T23:44:50.071960+0000","last_clean":"2024-04-24T23:44:50.071960+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:44:50.071960+0000","last_undegraded":"2024-04-24T23:44:50.071960+0000","last_fullsized":"2024-04-24T23:44:50.071960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072824+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:44:50.072824+0000","last_peered":"2024-04-24T23:44:50.072824+0000","last_clean":"2024-04-24T23:44:50.072824+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:44:50.072824+0000","last_undegraded":"2024-04-24T23:44:50.072824+0000","last_fullsized":"2024-04-24T23:44:50.072824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276286+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:44:50.276286+0000","last_peered":"2024-04-24T23:44:50.276286+0000","last_clean":"2024-04-24T23:44:50.276286+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:44:50.276286+0000","last_undegraded":"2024-04-24T23:44:50.276286+0000","last_fullsized":"2024-04-24T23:44:50.276286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276668+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:44:50.276668+0000","last_peered":"2024-04-24T23:44:50.276668+0000","last_clean":"2024-04-24T23:44:50.276668+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:44:50.276668+0000","last_undegraded":"2024-04-24T23:44:50.276668+0000","last_fullsized":"2024-04-24T23:44:50.276668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073870+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:44:50.073870+0000","last_peered":"2024-04-24T23:44:50.073870+0000","last_clean":"2024-04-24T23:44:50.073870+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:44:50.073870+0000","last_undegraded":"2024-04-24T23:44:50.073870+0000","last_fullsized":"2024-04-24T23:44:50.073870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190155+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:44:50.190155+0000","last_peered":"2024-04-24T23:44:50.190155+0000","last_clean":"2024-04-24T23:44:50.190155+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:44:50.190155+0000","last_undegraded":"2024-04-24T23:44:50.190155+0000","last_fullsized":"2024-04-24T23:44:50.190155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224849+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:44:50.224849+0000","last_peered":"2024-04-24T23:44:50.224849+0000","last_clean":"2024-04-24T23:44:50.224849+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:44:50.224849+0000","last_undegraded":"2024-04-24T23:44:50.224849+0000","last_fullsized":"2024-04-24T23:44:50.224849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078718+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:44:50.078718+0000","last_peered":"2024-04-24T23:44:50.078718+0000","last_clean":"2024-04-24T23:44:50.078718+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:44:50.078718+0000","last_undegraded":"2024-04-24T23:44:50.078718+0000","last_fullsized":"2024-04-24T23:44:50.078718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.277581+0000","last_change":"2024-04-24T23:44:50.277581+0000","last_active":"2024-04-24T23:44:50.277581+0000","last_peered":"2024-04-24T23:44:50.277581+0000","last_clean":"2024-04-24T23:44:50.277581+0000","last_became_active":"2024-04-24T23:44:50.277201+0000","last_became_peered":"2024-04-24T23:44:50.277201+0000","last_unstale":"2024-04-24T23:44:50.277581+0000","last_undegraded":"2024-04-24T23:44:50.277581+0000","last_fullsized":"2024-04-24T23:44:50.277581+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:58.685328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:14.257865+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:45:14.257865+0000","last_peered":"2024-04-24T23:45:14.257865+0000","last_clean":"2024-04-24T23:45:14.257865+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:45:14.257865+0000","last_undegraded":"2024-04-24T23:45:14.257865+0000","last_fullsized":"2024-04-24T23:45:14.257865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227525+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:44:50.227525+0000","last_peered":"2024-04-24T23:44:50.227525+0000","last_clean":"2024-04-24T23:44:50.227525+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:44:50.227525+0000","last_undegraded":"2024-04-24T23:44:50.227525+0000","last_fullsized":"2024-04-24T23:44:50.227525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071427+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:44:50.071427+0000","last_peered":"2024-04-24T23:44:50.071427+0000","last_clean":"2024-04-24T23:44:50.071427+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:44:50.071427+0000","last_undegraded":"2024-04-24T23:44:50.071427+0000","last_fullsized":"2024-04-24T23:44:50.071427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226823+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:44:50.226823+0000","last_peered":"2024-04-24T23:44:50.226823+0000","last_clean":"2024-04-24T23:44:50.226823+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:44:50.226823+0000","last_undegraded":"2024-04-24T23:44:50.226823+0000","last_fullsized":"2024-04-24T23:44:50.226823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276593+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:44:50.276593+0000","last_peered":"2024-04-24T23:44:50.276593+0000","last_clean":"2024-04-24T23:44:50.276593+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:44:50.276593+0000","last_undegraded":"2024-04-24T23:44:50.276593+0000","last_fullsized":"2024-04-24T23:44:50.276593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071625+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:44:50.071625+0000","last_peered":"2024-04-24T23:44:50.071625+0000","last_clean":"2024-04-24T23:44:50.071625+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:44:50.071625+0000","last_undegraded":"2024-04-24T23:44:50.071625+0000","last_fullsized":"2024-04-24T23:44:50.071625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074618+0000","last_change":"2024-04-24T23:43:49.235090+0000","last_active":"2024-04-24T23:44:50.074618+0000","last_peered":"2024-04-24T23:44:50.074618+0000","last_clean":"2024-04-24T23:44:50.074618+0000","last_became_active":"2024-04-24T23:43:49.223773+0000","last_became_peered":"2024-04-24T23:43:49.223773+0000","last_unstale":"2024-04-24T23:44:50.074618+0000","last_undegraded":"2024-04-24T23:44:50.074618+0000","last_fullsized":"2024-04-24T23:44:50.074618+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:23.776176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074166+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:44:50.074166+0000","last_peered":"2024-04-24T23:44:50.074166+0000","last_clean":"2024-04-24T23:44:50.074166+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:44:50.074166+0000","last_undegraded":"2024-04-24T23:44:50.074166+0000","last_fullsized":"2024-04-24T23:44:50.074166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071760+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:44:50.071760+0000","last_peered":"2024-04-24T23:44:50.071760+0000","last_clean":"2024-04-24T23:44:50.071760+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:44:50.071760+0000","last_undegraded":"2024-04-24T23:44:50.071760+0000","last_fullsized":"2024-04-24T23:44:50.071760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073467+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:44:50.073467+0000","last_peered":"2024-04-24T23:44:50.073467+0000","last_clean":"2024-04-24T23:44:50.073467+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:44:50.073467+0000","last_undegraded":"2024-04-24T23:44:50.073467+0000","last_fullsized":"2024-04-24T23:44:50.073467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071740+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:44:50.071740+0000","last_peered":"2024-04-24T23:44:50.071740+0000","last_clean":"2024-04-24T23:44:50.071740+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:44:50.071740+0000","last_undegraded":"2024-04-24T23:44:50.071740+0000","last_fullsized":"2024-04-24T23:44:50.071740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078499+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:44:50.078499+0000","last_peered":"2024-04-24T23:44:50.078499+0000","last_clean":"2024-04-24T23:44:50.078499+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:44:50.078499+0000","last_undegraded":"2024-04-24T23:44:50.078499+0000","last_fullsized":"2024-04-24T23:44:50.078499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:16.262165+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:45:16.262165+0000","last_peered":"2024-04-24T23:45:16.262165+0000","last_clean":"2024-04-24T23:45:16.262165+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:45:16.262165+0000","last_undegraded":"2024-04-24T23:45:16.262165+0000","last_fullsized":"2024-04-24T23:45:16.262165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073965+0000","last_change":"2024-04-24T23:43:49.224709+0000","last_active":"2024-04-24T23:44:50.073965+0000","last_peered":"2024-04-24T23:44:50.073965+0000","last_clean":"2024-04-24T23:44:50.073965+0000","last_became_active":"2024-04-24T23:43:49.224345+0000","last_became_peered":"2024-04-24T23:43:49.224345+0000","last_unstale":"2024-04-24T23:44:50.073965+0000","last_undegraded":"2024-04-24T23:44:50.073965+0000","last_fullsized":"2024-04-24T23:44:50.073965+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:56.093532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078701+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:44:50.078701+0000","last_peered":"2024-04-24T23:44:50.078701+0000","last_clean":"2024-04-24T23:44:50.078701+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:44:50.078701+0000","last_undegraded":"2024-04-24T23:44:50.078701+0000","last_fullsized":"2024-04-24T23:44:50.078701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226287+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:44:50.226287+0000","last_peered":"2024-04-24T23:44:50.226287+0000","last_clean":"2024-04-24T23:44:50.226287+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:44:50.226287+0000","last_undegraded":"2024-04-24T23:44:50.226287+0000","last_fullsized":"2024-04-24T23:44:50.226287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071791+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:44:50.071791+0000","last_peered":"2024-04-24T23:44:50.071791+0000","last_clean":"2024-04-24T23:44:50.071791+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:44:50.071791+0000","last_undegraded":"2024-04-24T23:44:50.071791+0000","last_fullsized":"2024-04-24T23:44:50.071791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190050+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:44:50.190050+0000","last_peered":"2024-04-24T23:44:50.190050+0000","last_clean":"2024-04-24T23:44:50.190050+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:44:50.190050+0000","last_undegraded":"2024-04-24T23:44:50.190050+0000","last_fullsized":"2024-04-24T23:44:50.190050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072614+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:44:50.072614+0000","last_peered":"2024-04-24T23:44:50.072614+0000","last_clean":"2024-04-24T23:44:50.072614+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:44:50.072614+0000","last_undegraded":"2024-04-24T23:44:50.072614+0000","last_fullsized":"2024-04-24T23:44:50.072614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224223+0000","last_change":"2024-04-24T23:43:49.226164+0000","last_active":"2024-04-24T23:44:50.224223+0000","last_peered":"2024-04-24T23:44:50.224223+0000","last_clean":"2024-04-24T23:44:50.224223+0000","last_became_active":"2024-04-24T23:43:49.225806+0000","last_became_peered":"2024-04-24T23:43:49.225806+0000","last_unstale":"2024-04-24T23:44:50.224223+0000","last_undegraded":"2024-04-24T23:44:50.224223+0000","last_fullsized":"2024-04-24T23:44:50.224223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:56.068947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276432+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:44:50.276432+0000","last_peered":"2024-04-24T23:44:50.276432+0000","last_clean":"2024-04-24T23:44:50.276432+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:44:50.276432+0000","last_undegraded":"2024-04-24T23:44:50.276432+0000","last_fullsized":"2024-04-24T23:44:50.276432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071398+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:44:50.071398+0000","last_peered":"2024-04-24T23:44:50.071398+0000","last_clean":"2024-04-24T23:44:50.071398+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:44:50.071398+0000","last_undegraded":"2024-04-24T23:44:50.071398+0000","last_fullsized":"2024-04-24T23:44:50.071398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224569+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:44:50.224569+0000","last_peered":"2024-04-24T23:44:50.224569+0000","last_clean":"2024-04-24T23:44:50.224569+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:44:50.224569+0000","last_undegraded":"2024-04-24T23:44:50.224569+0000","last_fullsized":"2024-04-24T23:44:50.224569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227216+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:44:50.227216+0000","last_peered":"2024-04-24T23:44:50.227216+0000","last_clean":"2024-04-24T23:44:50.227216+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:44:50.227216+0000","last_undegraded":"2024-04-24T23:44:50.227216+0000","last_fullsized":"2024-04-24T23:44:50.227216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.303688+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:45:15.303688+0000","last_peered":"2024-04-24T23:45:15.303688+0000","last_clean":"2024-04-24T23:45:15.303688+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:45:15.303688+0000","last_undegraded":"2024-04-24T23:45:15.303688+0000","last_fullsized":"2024-04-24T23:45:15.303688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074660+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:44:50.074660+0000","last_peered":"2024-04-24T23:44:50.074660+0000","last_clean":"2024-04-24T23:44:50.074660+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:44:50.074660+0000","last_undegraded":"2024-04-24T23:44:50.074660+0000","last_fullsized":"2024-04-24T23:44:50.074660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224398+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:44:50.224398+0000","last_peered":"2024-04-24T23:44:50.224398+0000","last_clean":"2024-04-24T23:44:50.224398+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:44:50.224398+0000","last_undegraded":"2024-04-24T23:44:50.224398+0000","last_fullsized":"2024-04-24T23:44:50.224398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228258+0000","last_change":"2024-04-24T23:43:49.226781+0000","last_active":"2024-04-24T23:44:50.228258+0000","last_peered":"2024-04-24T23:44:50.228258+0000","last_clean":"2024-04-24T23:44:50.228258+0000","last_became_active":"2024-04-24T23:43:49.222089+0000","last_became_peered":"2024-04-24T23:43:49.222089+0000","last_unstale":"2024-04-24T23:44:50.228258+0000","last_undegraded":"2024-04-24T23:44:50.228258+0000","last_fullsized":"2024-04-24T23:44:50.228258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:57.224621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276509+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:44:50.276509+0000","last_peered":"2024-04-24T23:44:50.276509+0000","last_clean":"2024-04-24T23:44:50.276509+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:44:50.276509+0000","last_undegraded":"2024-04-24T23:44:50.276509+0000","last_fullsized":"2024-04-24T23:44:50.276509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071804+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:44:50.071804+0000","last_peered":"2024-04-24T23:44:50.071804+0000","last_clean":"2024-04-24T23:44:50.071804+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:44:50.071804+0000","last_undegraded":"2024-04-24T23:44:50.071804+0000","last_fullsized":"2024-04-24T23:44:50.071804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071732+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:44:50.071732+0000","last_peered":"2024-04-24T23:44:50.071732+0000","last_clean":"2024-04-24T23:44:50.071732+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:44:50.071732+0000","last_undegraded":"2024-04-24T23:44:50.071732+0000","last_fullsized":"2024-04-24T23:44:50.071732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074559+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:44:50.074559+0000","last_peered":"2024-04-24T23:44:50.074559+0000","last_clean":"2024-04-24T23:44:50.074559+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:44:50.074559+0000","last_undegraded":"2024-04-24T23:44:50.074559+0000","last_fullsized":"2024-04-24T23:44:50.074559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224451+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:44:50.224451+0000","last_peered":"2024-04-24T23:44:50.224451+0000","last_clean":"2024-04-24T23:44:50.224451+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:44:50.224451+0000","last_undegraded":"2024-04-24T23:44:50.224451+0000","last_fullsized":"2024-04-24T23:44:50.224451+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'41","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.260922+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:45:15.260922+0000","last_peered":"2024-04-24T23:45:15.260922+0000","last_clean":"2024-04-24T23:45:15.260922+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:45:15.260922+0000","last_undegraded":"2024-04-24T23:45:15.260922+0000","last_fullsized":"2024-04-24T23:45:15.260922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68263,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":189,"num_write":43,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074503+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:44:50.074503+0000","last_peered":"2024-04-24T23:44:50.074503+0000","last_clean":"2024-04-24T23:44:50.074503+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:44:50.074503+0000","last_undegraded":"2024-04-24T23:44:50.074503+0000","last_fullsized":"2024-04-24T23:44:50.074503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072028+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:44:50.072028+0000","last_peered":"2024-04-24T23:44:50.072028+0000","last_clean":"2024-04-24T23:44:50.072028+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:44:50.072028+0000","last_undegraded":"2024-04-24T23:44:50.072028+0000","last_fullsized":"2024-04-24T23:44:50.072028+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078708+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:44:50.078708+0000","last_peered":"2024-04-24T23:44:50.078708+0000","last_clean":"2024-04-24T23:44:50.078708+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:44:50.078708+0000","last_undegraded":"2024-04-24T23:44:50.078708+0000","last_fullsized":"2024-04-24T23:44:50.078708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072796+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:44:50.072796+0000","last_peered":"2024-04-24T23:44:50.072796+0000","last_clean":"2024-04-24T23:44:50.072796+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:44:50.072796+0000","last_undegraded":"2024-04-24T23:44:50.072796+0000","last_fullsized":"2024-04-24T23:44:50.072796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189700+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:44:50.189700+0000","last_peered":"2024-04-24T23:44:50.189700+0000","last_clean":"2024-04-24T23:44:50.189700+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:44:50.189700+0000","last_undegraded":"2024-04-24T23:44:50.189700+0000","last_fullsized":"2024-04-24T23:44:50.189700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071783+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:44:50.071783+0000","last_peered":"2024-04-24T23:44:50.071783+0000","last_clean":"2024-04-24T23:44:50.071783+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:44:50.071783+0000","last_undegraded":"2024-04-24T23:44:50.071783+0000","last_fullsized":"2024-04-24T23:44:50.071783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228754+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:44:50.228754+0000","last_peered":"2024-04-24T23:44:50.228754+0000","last_clean":"2024-04-24T23:44:50.228754+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:44:50.228754+0000","last_undegraded":"2024-04-24T23:44:50.228754+0000","last_fullsized":"2024-04-24T23:44:50.228754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224947+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:44:50.224947+0000","last_peered":"2024-04-24T23:44:50.224947+0000","last_clean":"2024-04-24T23:44:50.224947+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:44:50.224947+0000","last_undegraded":"2024-04-24T23:44:50.224947+0000","last_fullsized":"2024-04-24T23:44:50.224947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.256361+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:45:12.256361+0000","last_peered":"2024-04-24T23:45:12.256361+0000","last_clean":"2024-04-24T23:45:12.256361+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:45:12.256361+0000","last_undegraded":"2024-04-24T23:45:12.256361+0000","last_fullsized":"2024-04-24T23:45:12.256361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228739+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:44:50.228739+0000","last_peered":"2024-04-24T23:44:50.228739+0000","last_clean":"2024-04-24T23:44:50.228739+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:44:50.228739+0000","last_undegraded":"2024-04-24T23:44:50.228739+0000","last_fullsized":"2024-04-24T23:44:50.228739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073186+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:44:50.073186+0000","last_peered":"2024-04-24T23:44:50.073186+0000","last_clean":"2024-04-24T23:44:50.073186+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:44:50.073186+0000","last_undegraded":"2024-04-24T23:44:50.073186+0000","last_fullsized":"2024-04-24T23:44:50.073186+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073794+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:44:50.073794+0000","last_peered":"2024-04-24T23:44:50.073794+0000","last_clean":"2024-04-24T23:44:50.073794+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:44:50.073794+0000","last_undegraded":"2024-04-24T23:44:50.073794+0000","last_fullsized":"2024-04-24T23:44:50.073794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071742+0000","last_change":"2024-04-24T23:43:49.223137+0000","last_active":"2024-04-24T23:44:50.071742+0000","last_peered":"2024-04-24T23:44:50.071742+0000","last_clean":"2024-04-24T23:44:50.071742+0000","last_became_active":"2024-04-24T23:43:49.222775+0000","last_became_peered":"2024-04-24T23:43:49.222775+0000","last_unstale":"2024-04-24T23:44:50.071742+0000","last_undegraded":"2024-04-24T23:44:50.071742+0000","last_fullsized":"2024-04-24T23:44:50.071742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:22.910320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071553+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:44:50.071553+0000","last_peered":"2024-04-24T23:44:50.071553+0000","last_clean":"2024-04-24T23:44:50.071553+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:44:50.071553+0000","last_undegraded":"2024-04-24T23:44:50.071553+0000","last_fullsized":"2024-04-24T23:44:50.071553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074870+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:44:50.074870+0000","last_peered":"2024-04-24T23:44:50.074870+0000","last_clean":"2024-04-24T23:44:50.074870+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:44:50.074870+0000","last_undegraded":"2024-04-24T23:44:50.074870+0000","last_fullsized":"2024-04-24T23:44:50.074870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190122+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:44:50.190122+0000","last_peered":"2024-04-24T23:44:50.190122+0000","last_clean":"2024-04-24T23:44:50.190122+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:44:50.190122+0000","last_undegraded":"2024-04-24T23:44:50.190122+0000","last_fullsized":"2024-04-24T23:44:50.190122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073941+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:44:50.073941+0000","last_peered":"2024-04-24T23:44:50.073941+0000","last_clean":"2024-04-24T23:44:50.073941+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:44:50.073941+0000","last_undegraded":"2024-04-24T23:44:50.073941+0000","last_fullsized":"2024-04-24T23:44:50.073941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224709+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:44:50.224709+0000","last_peered":"2024-04-24T23:44:50.224709+0000","last_clean":"2024-04-24T23:44:50.224709+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:44:50.224709+0000","last_undegraded":"2024-04-24T23:44:50.224709+0000","last_fullsized":"2024-04-24T23:44:50.224709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226822+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:44:50.226822+0000","last_peered":"2024-04-24T23:44:50.226822+0000","last_clean":"2024-04-24T23:44:50.226822+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:44:50.226822+0000","last_undegraded":"2024-04-24T23:44:50.226822+0000","last_fullsized":"2024-04-24T23:44:50.226822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072855+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:44:50.072855+0000","last_peered":"2024-04-24T23:44:50.072855+0000","last_clean":"2024-04-24T23:44:50.072855+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:44:50.072855+0000","last_undegraded":"2024-04-24T23:44:50.072855+0000","last_fullsized":"2024-04-24T23:44:50.072855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.257199+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:45:13.257199+0000","last_peered":"2024-04-24T23:45:13.257199+0000","last_clean":"2024-04-24T23:45:13.257199+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:45:13.257199+0000","last_undegraded":"2024-04-24T23:45:13.257199+0000","last_fullsized":"2024-04-24T23:45:13.257199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226654+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:44:50.226654+0000","last_peered":"2024-04-24T23:44:50.226654+0000","last_clean":"2024-04-24T23:44:50.226654+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:44:50.226654+0000","last_undegraded":"2024-04-24T23:44:50.226654+0000","last_fullsized":"2024-04-24T23:44:50.226654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073330+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:44:50.073330+0000","last_peered":"2024-04-24T23:44:50.073330+0000","last_clean":"2024-04-24T23:44:50.073330+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:44:50.073330+0000","last_undegraded":"2024-04-24T23:44:50.073330+0000","last_fullsized":"2024-04-24T23:44:50.073330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071894+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:44:50.071894+0000","last_peered":"2024-04-24T23:44:50.071894+0000","last_clean":"2024-04-24T23:44:50.071894+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:44:50.071894+0000","last_undegraded":"2024-04-24T23:44:50.071894+0000","last_fullsized":"2024-04-24T23:44:50.071894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.255804+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:45:12.255804+0000","last_peered":"2024-04-24T23:45:12.255804+0000","last_clean":"2024-04-24T23:45:12.255804+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:45:12.255804+0000","last_undegraded":"2024-04-24T23:45:12.255804+0000","last_fullsized":"2024-04-24T23:45:12.255804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228121+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:44:50.228121+0000","last_peered":"2024-04-24T23:44:50.228121+0000","last_clean":"2024-04-24T23:44:50.228121+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:44:50.228121+0000","last_undegraded":"2024-04-24T23:44:50.228121+0000","last_fullsized":"2024-04-24T23:44:50.228121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227286+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:44:50.227286+0000","last_peered":"2024-04-24T23:44:50.227286+0000","last_clean":"2024-04-24T23:44:50.227286+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:44:50.227286+0000","last_undegraded":"2024-04-24T23:44:50.227286+0000","last_fullsized":"2024-04-24T23:44:50.227286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071805+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:44:50.071805+0000","last_peered":"2024-04-24T23:44:50.071805+0000","last_clean":"2024-04-24T23:44:50.071805+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:44:50.071805+0000","last_undegraded":"2024-04-24T23:44:50.071805+0000","last_fullsized":"2024-04-24T23:44:50.071805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228428+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:44:50.228428+0000","last_peered":"2024-04-24T23:44:50.228428+0000","last_clean":"2024-04-24T23:44:50.228428+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:44:50.228428+0000","last_undegraded":"2024-04-24T23:44:50.228428+0000","last_fullsized":"2024-04-24T23:44:50.228428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072241+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:44:50.072241+0000","last_peered":"2024-04-24T23:44:50.072241+0000","last_clean":"2024-04-24T23:44:50.072241+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:44:50.072241+0000","last_undegraded":"2024-04-24T23:44:50.072241+0000","last_fullsized":"2024-04-24T23:44:50.072241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072552+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:44:50.072552+0000","last_peered":"2024-04-24T23:44:50.072552+0000","last_clean":"2024-04-24T23:44:50.072552+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:44:50.072552+0000","last_undegraded":"2024-04-24T23:44:50.072552+0000","last_fullsized":"2024-04-24T23:44:50.072552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227689+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:44:50.227689+0000","last_peered":"2024-04-24T23:44:50.227689+0000","last_clean":"2024-04-24T23:44:50.227689+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:44:50.227689+0000","last_undegraded":"2024-04-24T23:44:50.227689+0000","last_fullsized":"2024-04-24T23:44:50.227689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189830+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:44:50.189830+0000","last_peered":"2024-04-24T23:44:50.189830+0000","last_clean":"2024-04-24T23:44:50.189830+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:44:50.189830+0000","last_undegraded":"2024-04-24T23:44:50.189830+0000","last_fullsized":"2024-04-24T23:44:50.189830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189864+0000","last_change":"2024-04-24T23:43:49.235755+0000","last_active":"2024-04-24T23:44:50.189864+0000","last_peered":"2024-04-24T23:44:50.189864+0000","last_clean":"2024-04-24T23:44:50.189864+0000","last_became_active":"2024-04-24T23:43:49.230488+0000","last_became_peered":"2024-04-24T23:43:49.230488+0000","last_unstale":"2024-04-24T23:44:50.189864+0000","last_undegraded":"2024-04-24T23:44:50.189864+0000","last_fullsized":"2024-04-24T23:44:50.189864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:12.034199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073003+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:44:50.073003+0000","last_peered":"2024-04-24T23:44:50.073003+0000","last_clean":"2024-04-24T23:44:50.073003+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:44:50.073003+0000","last_undegraded":"2024-04-24T23:44:50.073003+0000","last_fullsized":"2024-04-24T23:44:50.073003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227292+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:44:50.227292+0000","last_peered":"2024-04-24T23:44:50.227292+0000","last_clean":"2024-04-24T23:44:50.227292+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:44:50.227292+0000","last_undegraded":"2024-04-24T23:44:50.227292+0000","last_fullsized":"2024-04-24T23:44:50.227292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:01.190467+0000","last_change":"2024-04-24T23:43:49.236439+0000","last_active":"2024-04-24T23:45:01.190467+0000","last_peered":"2024-04-24T23:45:01.190467+0000","last_clean":"2024-04-24T23:45:01.190467+0000","last_became_active":"2024-04-24T23:43:49.227756+0000","last_became_peered":"2024-04-24T23:43:49.227756+0000","last_unstale":"2024-04-24T23:45:01.190467+0000","last_undegraded":"2024-04-24T23:45:01.190467+0000","last_fullsized":"2024-04-24T23:45:01.190467+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:34.790165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227799+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:44:50.227799+0000","last_peered":"2024-04-24T23:44:50.227799+0000","last_clean":"2024-04-24T23:44:50.227799+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:44:50.227799+0000","last_undegraded":"2024-04-24T23:44:50.227799+0000","last_fullsized":"2024-04-24T23:44:50.227799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078047+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:44:50.078047+0000","last_peered":"2024-04-24T23:44:50.078047+0000","last_clean":"2024-04-24T23:44:50.078047+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:44:50.078047+0000","last_undegraded":"2024-04-24T23:44:50.078047+0000","last_fullsized":"2024-04-24T23:44:50.078047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071411+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:44:50.071411+0000","last_peered":"2024-04-24T23:44:50.071411+0000","last_clean":"2024-04-24T23:44:50.071411+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:44:50.071411+0000","last_undegraded":"2024-04-24T23:44:50.071411+0000","last_fullsized":"2024-04-24T23:44:50.071411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276642+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:44:50.276642+0000","last_peered":"2024-04-24T23:44:50.276642+0000","last_clean":"2024-04-24T23:44:50.276642+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:44:50.276642+0000","last_undegraded":"2024-04-24T23:44:50.276642+0000","last_fullsized":"2024-04-24T23:44:50.276642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228660+0000","last_change":"2024-04-24T23:44:50.228080+0000","last_active":"2024-04-24T23:44:50.228660+0000","last_peered":"2024-04-24T23:44:50.228660+0000","last_clean":"2024-04-24T23:44:50.228660+0000","last_became_active":"2024-04-24T23:44:50.227599+0000","last_became_peered":"2024-04-24T23:44:50.227599+0000","last_unstale":"2024-04-24T23:44:50.228660+0000","last_undegraded":"2024-04-24T23:44:50.228660+0000","last_fullsized":"2024-04-24T23:44:50.228660+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:49.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074277+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:44:50.074277+0000","last_peered":"2024-04-24T23:44:50.074277+0000","last_clean":"2024-04-24T23:44:50.074277+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:44:50.074277+0000","last_undegraded":"2024-04-24T23:44:50.074277+0000","last_fullsized":"2024-04-24T23:44:50.074277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073609+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:44:50.073609+0000","last_peered":"2024-04-24T23:44:50.073609+0000","last_clean":"2024-04-24T23:44:50.073609+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:44:50.073609+0000","last_undegraded":"2024-04-24T23:44:50.073609+0000","last_fullsized":"2024-04-24T23:44:50.073609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.229292+0000","last_change":"2024-04-24T23:44:50.229292+0000","last_active":"2024-04-24T23:44:50.229292+0000","last_peered":"2024-04-24T23:44:50.229292+0000","last_clean":"2024-04-24T23:44:50.229292+0000","last_became_active":"2024-04-24T23:44:50.228528+0000","last_became_peered":"2024-04-24T23:44:50.228528+0000","last_unstale":"2024-04-24T23:44:50.229292+0000","last_undegraded":"2024-04-24T23:44:50.229292+0000","last_fullsized":"2024-04-24T23:44:50.229292+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:53.700501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227345+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:44:50.227345+0000","last_peered":"2024-04-24T23:44:50.227345+0000","last_clean":"2024-04-24T23:44:50.227345+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:44:50.227345+0000","last_undegraded":"2024-04-24T23:44:50.227345+0000","last_fullsized":"2024-04-24T23:44:50.227345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189662+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:44:50.189662+0000","last_peered":"2024-04-24T23:44:50.189662+0000","last_clean":"2024-04-24T23:44:50.189662+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:44:50.189662+0000","last_undegraded":"2024-04-24T23:44:50.189662+0000","last_fullsized":"2024-04-24T23:44:50.189662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071954+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:44:50.071954+0000","last_peered":"2024-04-24T23:44:50.071954+0000","last_clean":"2024-04-24T23:44:50.071954+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:44:50.071954+0000","last_undegraded":"2024-04-24T23:44:50.071954+0000","last_fullsized":"2024-04-24T23:44:50.071954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.077868+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:44:50.077868+0000","last_peered":"2024-04-24T23:44:50.077868+0000","last_clean":"2024-04-24T23:44:50.077868+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:44:50.077868+0000","last_undegraded":"2024-04-24T23:44:50.077868+0000","last_fullsized":"2024-04-24T23:44:50.077868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227346+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:44:50.227346+0000","last_peered":"2024-04-24T23:44:50.227346+0000","last_clean":"2024-04-24T23:44:50.227346+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:44:50.227346+0000","last_undegraded":"2024-04-24T23:44:50.227346+0000","last_fullsized":"2024-04-24T23:44:50.227346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072175+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:44:50.072175+0000","last_peered":"2024-04-24T23:44:50.072175+0000","last_clean":"2024-04-24T23:44:50.072175+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:44:50.072175+0000","last_undegraded":"2024-04-24T23:44:50.072175+0000","last_fullsized":"2024-04-24T23:44:50.072175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074163+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:44:50.074163+0000","last_peered":"2024-04-24T23:44:50.074163+0000","last_clean":"2024-04-24T23:44:50.074163+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:44:50.074163+0000","last_undegraded":"2024-04-24T23:44:50.074163+0000","last_fullsized":"2024-04-24T23:44:50.074163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228420+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:44:50.228420+0000","last_peered":"2024-04-24T23:44:50.228420+0000","last_clean":"2024-04-24T23:44:50.228420+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:44:50.228420+0000","last_undegraded":"2024-04-24T23:44:50.228420+0000","last_fullsized":"2024-04-24T23:44:50.228420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.230519+0000","last_change":"2024-04-24T23:44:50.230008+0000","last_active":"2024-04-24T23:44:50.230519+0000","last_peered":"2024-04-24T23:44:50.230519+0000","last_clean":"2024-04-24T23:44:50.230519+0000","last_became_active":"2024-04-24T23:44:50.229452+0000","last_became_peered":"2024-04-24T23:44:50.229452+0000","last_unstale":"2024-04-24T23:44:50.230519+0000","last_undegraded":"2024-04-24T23:44:50.230519+0000","last_fullsized":"2024-04-24T23:44:50.230519+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:13.691069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.300487+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:45:13.300487+0000","last_peered":"2024-04-24T23:45:13.300487+0000","last_clean":"2024-04-24T23:45:13.300487+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:45:13.300487+0000","last_undegraded":"2024-04-24T23:45:13.300487+0000","last_fullsized":"2024-04-24T23:45:13.300487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1478,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228299+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:44:50.228299+0000","last_peered":"2024-04-24T23:44:50.228299+0000","last_clean":"2024-04-24T23:44:50.228299+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:44:50.228299+0000","last_undegraded":"2024-04-24T23:44:50.228299+0000","last_fullsized":"2024-04-24T23:44:50.228299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276473+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:44:50.276473+0000","last_peered":"2024-04-24T23:44:50.276473+0000","last_clean":"2024-04-24T23:44:50.276473+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:44:50.276473+0000","last_undegraded":"2024-04-24T23:44:50.276473+0000","last_fullsized":"2024-04-24T23:44:50.276473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071951+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:44:50.071951+0000","last_peered":"2024-04-24T23:44:50.071951+0000","last_clean":"2024-04-24T23:44:50.071951+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:44:50.071951+0000","last_undegraded":"2024-04-24T23:44:50.071951+0000","last_fullsized":"2024-04-24T23:44:50.071951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228018+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:44:50.228018+0000","last_peered":"2024-04-24T23:44:50.228018+0000","last_clean":"2024-04-24T23:44:50.228018+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:44:50.228018+0000","last_undegraded":"2024-04-24T23:44:50.228018+0000","last_fullsized":"2024-04-24T23:44:50.228018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071162+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:44:50.071162+0000","last_peered":"2024-04-24T23:44:50.071162+0000","last_clean":"2024-04-24T23:44:50.071162+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:44:50.071162+0000","last_undegraded":"2024-04-24T23:44:50.071162+0000","last_fullsized":"2024-04-24T23:44:50.071162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073134+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:44:50.073134+0000","last_peered":"2024-04-24T23:44:50.073134+0000","last_clean":"2024-04-24T23:44:50.073134+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:44:50.073134+0000","last_undegraded":"2024-04-24T23:44:50.073134+0000","last_fullsized":"2024-04-24T23:44:50.073134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228522+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:44:50.228522+0000","last_peered":"2024-04-24T23:44:50.228522+0000","last_clean":"2024-04-24T23:44:50.228522+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:44:50.228522+0000","last_undegraded":"2024-04-24T23:44:50.228522+0000","last_fullsized":"2024-04-24T23:44:50.228522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189686+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:44:50.189686+0000","last_peered":"2024-04-24T23:44:50.189686+0000","last_clean":"2024-04-24T23:44:50.189686+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:44:50.189686+0000","last_undegraded":"2024-04-24T23:44:50.189686+0000","last_fullsized":"2024-04-24T23:44:50.189686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078449+0000","last_change":"2024-04-24T23:43:49.223272+0000","last_active":"2024-04-24T23:44:50.078449+0000","last_peered":"2024-04-24T23:44:50.078449+0000","last_clean":"2024-04-24T23:44:50.078449+0000","last_became_active":"2024-04-24T23:43:49.222921+0000","last_became_peered":"2024-04-24T23:43:49.222921+0000","last_unstale":"2024-04-24T23:44:50.078449+0000","last_undegraded":"2024-04-24T23:44:50.078449+0000","last_fullsized":"2024-04-24T23:44:50.078449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:10.443359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078022+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:44:50.078022+0000","last_peered":"2024-04-24T23:44:50.078022+0000","last_clean":"2024-04-24T23:44:50.078022+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:44:50.078022+0000","last_undegraded":"2024-04-24T23:44:50.078022+0000","last_fullsized":"2024-04-24T23:44:50.078022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276474+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:44:50.276474+0000","last_peered":"2024-04-24T23:44:50.276474+0000","last_clean":"2024-04-24T23:44:50.276474+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:44:50.276474+0000","last_undegraded":"2024-04-24T23:44:50.276474+0000","last_fullsized":"2024-04-24T23:44:50.276474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072969+0000","last_change":"2024-04-24T23:43:49.230897+0000","last_active":"2024-04-24T23:44:50.072969+0000","last_peered":"2024-04-24T23:44:50.072969+0000","last_clean":"2024-04-24T23:44:50.072969+0000","last_became_active":"2024-04-24T23:43:49.230514+0000","last_became_peered":"2024-04-24T23:43:49.230514+0000","last_unstale":"2024-04-24T23:44:50.072969+0000","last_undegraded":"2024-04-24T23:44:50.072969+0000","last_fullsized":"2024-04-24T23:44:50.072969+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:24.826152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190172+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:44:50.190172+0000","last_peered":"2024-04-24T23:44:50.190172+0000","last_clean":"2024-04-24T23:44:50.190172+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:44:50.190172+0000","last_undegraded":"2024-04-24T23:44:50.190172+0000","last_fullsized":"2024-04-24T23:44:50.190172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.079053+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:44:50.079053+0000","last_peered":"2024-04-24T23:44:50.079053+0000","last_clean":"2024-04-24T23:44:50.079053+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:44:50.079053+0000","last_undegraded":"2024-04-24T23:44:50.079053+0000","last_fullsized":"2024-04-24T23:44:50.079053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227836+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:44:50.227836+0000","last_peered":"2024-04-24T23:44:50.227836+0000","last_clean":"2024-04-24T23:44:50.227836+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:44:50.227836+0000","last_undegraded":"2024-04-24T23:44:50.227836+0000","last_fullsized":"2024-04-24T23:44:50.227836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071656+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:44:50.071656+0000","last_peered":"2024-04-24T23:44:50.071656+0000","last_clean":"2024-04-24T23:44:50.071656+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:44:50.071656+0000","last_undegraded":"2024-04-24T23:44:50.071656+0000","last_fullsized":"2024-04-24T23:44:50.071656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227382+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:44:50.227382+0000","last_peered":"2024-04-24T23:44:50.227382+0000","last_clean":"2024-04-24T23:44:50.227382+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:44:50.227382+0000","last_undegraded":"2024-04-24T23:44:50.227382+0000","last_fullsized":"2024-04-24T23:44:50.227382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.075088+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:44:50.075088+0000","last_peered":"2024-04-24T23:44:50.075088+0000","last_clean":"2024-04-24T23:44:50.075088+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:44:50.075088+0000","last_undegraded":"2024-04-24T23:44:50.075088+0000","last_fullsized":"2024-04-24T23:44:50.075088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073092+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:44:50.073092+0000","last_peered":"2024-04-24T23:44:50.073092+0000","last_clean":"2024-04-24T23:44:50.073092+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:44:50.073092+0000","last_undegraded":"2024-04-24T23:44:50.073092+0000","last_fullsized":"2024-04-24T23:44:50.073092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189908+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:44:50.189908+0000","last_peered":"2024-04-24T23:44:50.189908+0000","last_clean":"2024-04-24T23:44:50.189908+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:44:50.189908+0000","last_undegraded":"2024-04-24T23:44:50.189908+0000","last_fullsized":"2024-04-24T23:44:50.189908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75525,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1205,"num_read_kb":685,"num_write":255,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":112,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":221202,"data_compressed":4866,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520817,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1176008,"data_compressed":64433,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":524423,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":520593,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":1117343,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515831,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":585124,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517595,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":580279,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":515539,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520437,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1112393,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":70015,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65170,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70531,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:45:19.223 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-24T23:45:19.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 -- 172.21.15.107:0/1771950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88981033f0 msgr2=0x7f8898109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 --2- 172.21.15.107:0/1771950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88981033f0 0x7f8898109c80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8890009cd0 tx=0x7f889002a8e0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 -- 172.21.15.107:0/1771950638 shutdown_connections 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 --2- 172.21.15.107:0/1771950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88981033f0 0x7f8898109c80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 --2- 172.21.15.107:0/1771950638 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8898102a30 0x7f8898102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 --2- 172.21.15.107:0/1771950638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 0x7f88980ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 -- 172.21.15.107:0/1771950638 >> 172.21.15.107:0/1771950638 conn(0x7f88980fb3b0 msgr2=0x7f88980fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.362+0000 7f889d734640 1 -- 172.21.15.107:0/1771950638 shutdown_connections 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 -- 172.21.15.107:0/1771950638 wait complete. 2024-04-24T23:45:19.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 Processor -- start 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 -- start start 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 0x7f8898197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 0x7f889819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f889810d440 con 0x7f8898102a30 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f889810d2c0 con 0x7f88980ffa60 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f889d734640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f889810d5c0 con 0x7f88981033f0 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.363+0000 7f88967fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42962/0 (socket says 172.21.15.107:42962) 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 -- 172.21.15.107:0/2589978578 learned_addr learned my addr 172.21.15.107:0/2589978578 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88977fe640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 0x7f889819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f8896ffd640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 0x7f8898197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 -- 172.21.15.107:0/2589978578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 msgr2=0x7f889819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 0x7f889819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 -- 172.21.15.107:0/2589978578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 msgr2=0x7f8898197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 0x7f8898197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 -- 172.21.15.107:0/2589978578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889819f6b0 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88977fe640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 0x7f889819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f88967fc640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f888800e9f0 tx=0x7f888800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888800cd80 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889819f9a0 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8888005500 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888010690 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.364+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889819fea0 con 0x7f8898102a30 2024-04-24T23:45:19.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.365+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8898108b60 con 0x7f8898102a30 2024-04-24T23:45:19.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.366+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f88880046c0 con 0x7f8898102a30 2024-04-24T23:45:19.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.367+0000 7f887ffff640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 0x7f88800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.367+0000 7f8896ffd640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 0x7f88800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.367+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f88880cf8e0 con 0x7f8898102a30 2024-04-24T23:45:19.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.367+0000 7f8896ffd640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 0x7f88800a5050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8898069bc0 tx=0x7f888c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.371+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f888809c470 con 0x7f8898102a30 2024-04-24T23:45:19.398 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:19 smithi195 ceph-mon[33019]: pgmap v368: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:19.398 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3954708591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:19.398 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:19 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.398 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:19 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2095203921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:19 smithi107 ceph-mon[29118]: pgmap v368: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3954708591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:19 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:19 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2095203921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:19 smithi170 ceph-mon[32714]: pgmap v368: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:19.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3954708591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:19.407 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:19 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.407 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:19 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2095203921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.548+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 --> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8898061b80 con 0x7f88800a2b90 2024-04-24T23:45:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.558+0000 7f887ffff640 1 -- 172.21.15.107:0/2589978578 <== mgr.14152 v2:172.21.15.107:6800/1159279165 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340833 (secure 0 0 0) 0x7f8898061b80 con 0x7f88800a2b90 2024-04-24T23:45:19.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:19.563 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 msgr2=0x7f88800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 0x7f88800a5050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8898069bc0 tx=0x7f888c00b390 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 msgr2=0x7f8898197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f888800e9f0 tx=0x7f888800edf0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 shutdown_connections 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f88800a2b90 0x7f88800a5050 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f88981033f0 0x7f889819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8898102a30 0x7f8898197fd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 --2- 172.21.15.107:0/2589978578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88980ffa60 0x7f8898197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 >> 172.21.15.107:0/2589978578 conn(0x7f88980fb3b0 msgr2=0x7f8898105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 shutdown_connections 2024-04-24T23:45:19.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.561+0000 7f889d734640 1 -- 172.21.15.107:0/2589978578 wait complete. 2024-04-24T23:45:19.572 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-24T23:45:18.013428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666127,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1331,"num_read_kb":794,"num_write":438,"num_write_kb":3096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1180848,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365964,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575860,"statfs":{"total":1151940427776,"available":1151565680640,"internally_reserved":0,"allocated":12894208,"data_stored":8206382,"data_compressed":432924,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":62,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003398"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190209+0000","last_change":"2024-04-24T23:42:51.075090+0000","last_active":"2024-04-24T23:44:50.190209+0000","last_peered":"2024-04-24T23:44:50.190209+0000","last_clean":"2024-04-24T23:44:50.190209+0000","last_became_active":"2024-04-24T23:42:51.073861+0000","last_became_peered":"2024-04-24T23:42:51.073861+0000","last_unstale":"2024-04-24T23:44:50.190209+0000","last_undegraded":"2024-04-24T23:44:50.190209+0000","last_fullsized":"2024-04-24T23:44:50.190209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:38.646633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276613+0000","last_change":"2024-04-24T23:42:51.076255+0000","last_active":"2024-04-24T23:44:50.276613+0000","last_peered":"2024-04-24T23:44:50.276613+0000","last_clean":"2024-04-24T23:44:50.276613+0000","last_became_active":"2024-04-24T23:42:51.075526+0000","last_became_peered":"2024-04-24T23:42:51.075526+0000","last_unstale":"2024-04-24T23:44:50.276613+0000","last_undegraded":"2024-04-24T23:44:50.276613+0000","last_fullsized":"2024-04-24T23:44:50.276613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:44:56.301587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276502+0000","last_change":"2024-04-24T23:42:51.079546+0000","last_active":"2024-04-24T23:44:50.276502+0000","last_peered":"2024-04-24T23:44:50.276502+0000","last_clean":"2024-04-24T23:44:50.276502+0000","last_became_active":"2024-04-24T23:42:51.079178+0000","last_became_peered":"2024-04-24T23:42:51.079178+0000","last_unstale":"2024-04-24T23:44:50.276502+0000","last_undegraded":"2024-04-24T23:44:50.276502+0000","last_fullsized":"2024-04-24T23:44:50.276502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:48.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072084+0000","last_change":"2024-04-24T23:42:51.078884+0000","last_active":"2024-04-24T23:44:50.072084+0000","last_peered":"2024-04-24T23:44:50.072084+0000","last_clean":"2024-04-24T23:44:50.072084+0000","last_became_active":"2024-04-24T23:42:51.078335+0000","last_became_peered":"2024-04-24T23:42:51.078335+0000","last_unstale":"2024-04-24T23:44:50.072084+0000","last_undegraded":"2024-04-24T23:44:50.072084+0000","last_fullsized":"2024-04-24T23:44:50.072084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:42.186032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073969+0000","last_change":"2024-04-24T23:42:51.070269+0000","last_active":"2024-04-24T23:44:50.073969+0000","last_peered":"2024-04-24T23:44:50.073969+0000","last_clean":"2024-04-24T23:44:50.073969+0000","last_became_active":"2024-04-24T23:42:51.069564+0000","last_became_peered":"2024-04-24T23:42:51.069564+0000","last_unstale":"2024-04-24T23:44:50.073969+0000","last_undegraded":"2024-04-24T23:44:50.073969+0000","last_fullsized":"2024-04-24T23:44:50.073969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:02.246400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226383+0000","last_change":"2024-04-24T23:42:51.079210+0000","last_active":"2024-04-24T23:44:50.226383+0000","last_peered":"2024-04-24T23:44:50.226383+0000","last_clean":"2024-04-24T23:44:50.226383+0000","last_became_active":"2024-04-24T23:42:51.078883+0000","last_became_peered":"2024-04-24T23:42:51.078883+0000","last_unstale":"2024-04-24T23:44:50.226383+0000","last_undegraded":"2024-04-24T23:44:50.226383+0000","last_fullsized":"2024-04-24T23:44:50.226383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:44:35.045875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227750+0000","last_change":"2024-04-24T23:42:51.073970+0000","last_active":"2024-04-24T23:44:50.227750+0000","last_peered":"2024-04-24T23:44:50.227750+0000","last_clean":"2024-04-24T23:44:50.227750+0000","last_became_active":"2024-04-24T23:42:51.073497+0000","last_became_peered":"2024-04-24T23:42:51.073497+0000","last_unstale":"2024-04-24T23:44:50.227750+0000","last_undegraded":"2024-04-24T23:44:50.227750+0000","last_fullsized":"2024-04-24T23:44:50.227750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:11.276147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227133+0000","last_change":"2024-04-24T23:42:51.082695+0000","last_active":"2024-04-24T23:44:50.227133+0000","last_peered":"2024-04-24T23:44:50.227133+0000","last_clean":"2024-04-24T23:44:50.227133+0000","last_became_active":"2024-04-24T23:42:51.082381+0000","last_became_peered":"2024-04-24T23:42:51.082381+0000","last_unstale":"2024-04-24T23:44:50.227133+0000","last_undegraded":"2024-04-24T23:44:50.227133+0000","last_fullsized":"2024-04-24T23:44:50.227133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:02.116227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224664+0000","last_change":"2024-04-24T23:43:49.227407+0000","last_active":"2024-04-24T23:44:50.224664+0000","last_peered":"2024-04-24T23:44:50.224664+0000","last_clean":"2024-04-24T23:44:50.224664+0000","last_became_active":"2024-04-24T23:43:49.227076+0000","last_became_peered":"2024-04-24T23:43:49.227076+0000","last_unstale":"2024-04-24T23:44:50.224664+0000","last_undegraded":"2024-04-24T23:44:50.224664+0000","last_fullsized":"2024-04-24T23:44:50.224664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:24.426521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227028+0000","last_change":"2024-04-24T23:42:51.079792+0000","last_active":"2024-04-24T23:44:50.227028+0000","last_peered":"2024-04-24T23:44:50.227028+0000","last_clean":"2024-04-24T23:44:50.227028+0000","last_became_active":"2024-04-24T23:42:51.079487+0000","last_became_peered":"2024-04-24T23:42:51.079487+0000","last_unstale":"2024-04-24T23:44:50.227028+0000","last_undegraded":"2024-04-24T23:44:50.227028+0000","last_fullsized":"2024-04-24T23:44:50.227028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.498804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228179+0000","last_change":"2024-04-24T23:42:51.080640+0000","last_active":"2024-04-24T23:44:50.228179+0000","last_peered":"2024-04-24T23:44:50.228179+0000","last_clean":"2024-04-24T23:44:50.228179+0000","last_became_active":"2024-04-24T23:42:51.080287+0000","last_became_peered":"2024-04-24T23:42:51.080287+0000","last_unstale":"2024-04-24T23:44:50.228179+0000","last_undegraded":"2024-04-24T23:44:50.228179+0000","last_fullsized":"2024-04-24T23:44:50.228179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:34.210683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071960+0000","last_change":"2024-04-24T23:42:51.079706+0000","last_active":"2024-04-24T23:44:50.071960+0000","last_peered":"2024-04-24T23:44:50.071960+0000","last_clean":"2024-04-24T23:44:50.071960+0000","last_became_active":"2024-04-24T23:42:51.079219+0000","last_became_peered":"2024-04-24T23:42:51.079219+0000","last_unstale":"2024-04-24T23:44:50.071960+0000","last_undegraded":"2024-04-24T23:44:50.071960+0000","last_fullsized":"2024-04-24T23:44:50.071960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:45.931842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072824+0000","last_change":"2024-04-24T23:42:51.058948+0000","last_active":"2024-04-24T23:44:50.072824+0000","last_peered":"2024-04-24T23:44:50.072824+0000","last_clean":"2024-04-24T23:44:50.072824+0000","last_became_active":"2024-04-24T23:42:51.058597+0000","last_became_peered":"2024-04-24T23:42:51.058597+0000","last_unstale":"2024-04-24T23:44:50.072824+0000","last_undegraded":"2024-04-24T23:44:50.072824+0000","last_fullsized":"2024-04-24T23:44:50.072824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:43.189417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276286+0000","last_change":"2024-04-24T23:42:51.076839+0000","last_active":"2024-04-24T23:44:50.276286+0000","last_peered":"2024-04-24T23:44:50.276286+0000","last_clean":"2024-04-24T23:44:50.276286+0000","last_became_active":"2024-04-24T23:42:51.075955+0000","last_became_peered":"2024-04-24T23:42:51.075955+0000","last_unstale":"2024-04-24T23:44:50.276286+0000","last_undegraded":"2024-04-24T23:44:50.276286+0000","last_fullsized":"2024-04-24T23:44:50.276286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:43.591023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276668+0000","last_change":"2024-04-24T23:42:51.077240+0000","last_active":"2024-04-24T23:44:50.276668+0000","last_peered":"2024-04-24T23:44:50.276668+0000","last_clean":"2024-04-24T23:44:50.276668+0000","last_became_active":"2024-04-24T23:42:51.076672+0000","last_became_peered":"2024-04-24T23:42:51.076672+0000","last_unstale":"2024-04-24T23:44:50.276668+0000","last_undegraded":"2024-04-24T23:44:50.276668+0000","last_fullsized":"2024-04-24T23:44:50.276668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:29:45.469647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073870+0000","last_change":"2024-04-24T23:42:51.075808+0000","last_active":"2024-04-24T23:44:50.073870+0000","last_peered":"2024-04-24T23:44:50.073870+0000","last_clean":"2024-04-24T23:44:50.073870+0000","last_became_active":"2024-04-24T23:42:51.075468+0000","last_became_peered":"2024-04-24T23:42:51.075468+0000","last_unstale":"2024-04-24T23:44:50.073870+0000","last_undegraded":"2024-04-24T23:44:50.073870+0000","last_fullsized":"2024-04-24T23:44:50.073870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:36.170135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190155+0000","last_change":"2024-04-24T23:42:51.075620+0000","last_active":"2024-04-24T23:44:50.190155+0000","last_peered":"2024-04-24T23:44:50.190155+0000","last_clean":"2024-04-24T23:44:50.190155+0000","last_became_active":"2024-04-24T23:42:51.074321+0000","last_became_peered":"2024-04-24T23:42:51.074321+0000","last_unstale":"2024-04-24T23:44:50.190155+0000","last_undegraded":"2024-04-24T23:44:50.190155+0000","last_fullsized":"2024-04-24T23:44:50.190155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:17.192264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224849+0000","last_change":"2024-04-24T23:42:51.075791+0000","last_active":"2024-04-24T23:44:50.224849+0000","last_peered":"2024-04-24T23:44:50.224849+0000","last_clean":"2024-04-24T23:44:50.224849+0000","last_became_active":"2024-04-24T23:42:51.075456+0000","last_became_peered":"2024-04-24T23:42:51.075456+0000","last_unstale":"2024-04-24T23:44:50.224849+0000","last_undegraded":"2024-04-24T23:44:50.224849+0000","last_fullsized":"2024-04-24T23:44:50.224849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:36.382606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078718+0000","last_change":"2024-04-24T23:42:51.074079+0000","last_active":"2024-04-24T23:44:50.078718+0000","last_peered":"2024-04-24T23:44:50.078718+0000","last_clean":"2024-04-24T23:44:50.078718+0000","last_became_active":"2024-04-24T23:42:51.073689+0000","last_became_peered":"2024-04-24T23:42:51.073689+0000","last_unstale":"2024-04-24T23:44:50.078718+0000","last_undegraded":"2024-04-24T23:44:50.078718+0000","last_fullsized":"2024-04-24T23:44:50.078718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:01.996616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.277581+0000","last_change":"2024-04-24T23:44:50.277581+0000","last_active":"2024-04-24T23:44:50.277581+0000","last_peered":"2024-04-24T23:44:50.277581+0000","last_clean":"2024-04-24T23:44:50.277581+0000","last_became_active":"2024-04-24T23:44:50.277201+0000","last_became_peered":"2024-04-24T23:44:50.277201+0000","last_unstale":"2024-04-24T23:44:50.277581+0000","last_undegraded":"2024-04-24T23:44:50.277581+0000","last_fullsized":"2024-04-24T23:44:50.277581+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:58.685328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:14.257865+0000","last_change":"2024-04-24T23:42:50.068698+0000","last_active":"2024-04-24T23:45:14.257865+0000","last_peered":"2024-04-24T23:45:14.257865+0000","last_clean":"2024-04-24T23:45:14.257865+0000","last_became_active":"2024-04-24T23:42:50.068346+0000","last_became_peered":"2024-04-24T23:42:50.068346+0000","last_unstale":"2024-04-24T23:45:14.257865+0000","last_undegraded":"2024-04-24T23:45:14.257865+0000","last_fullsized":"2024-04-24T23:45:14.257865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:47.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227525+0000","last_change":"2024-04-24T23:42:51.074227+0000","last_active":"2024-04-24T23:44:50.227525+0000","last_peered":"2024-04-24T23:44:50.227525+0000","last_clean":"2024-04-24T23:44:50.227525+0000","last_became_active":"2024-04-24T23:42:51.073890+0000","last_became_peered":"2024-04-24T23:42:51.073890+0000","last_unstale":"2024-04-24T23:44:50.227525+0000","last_undegraded":"2024-04-24T23:44:50.227525+0000","last_fullsized":"2024-04-24T23:44:50.227525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:36.358866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071427+0000","last_change":"2024-04-24T23:42:55.323575+0000","last_active":"2024-04-24T23:44:50.071427+0000","last_peered":"2024-04-24T23:44:50.071427+0000","last_clean":"2024-04-24T23:44:50.071427+0000","last_became_active":"2024-04-24T23:42:55.323162+0000","last_became_peered":"2024-04-24T23:42:55.323162+0000","last_unstale":"2024-04-24T23:44:50.071427+0000","last_undegraded":"2024-04-24T23:44:50.071427+0000","last_fullsized":"2024-04-24T23:44:50.071427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:35.618039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226823+0000","last_change":"2024-04-24T23:42:50.076393+0000","last_active":"2024-04-24T23:44:50.226823+0000","last_peered":"2024-04-24T23:44:50.226823+0000","last_clean":"2024-04-24T23:44:50.226823+0000","last_became_active":"2024-04-24T23:42:50.076037+0000","last_became_peered":"2024-04-24T23:42:50.076037+0000","last_unstale":"2024-04-24T23:44:50.226823+0000","last_undegraded":"2024-04-24T23:44:50.226823+0000","last_fullsized":"2024-04-24T23:44:50.226823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:16.472600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276593+0000","last_change":"2024-04-24T23:42:51.072965+0000","last_active":"2024-04-24T23:44:50.276593+0000","last_peered":"2024-04-24T23:44:50.276593+0000","last_clean":"2024-04-24T23:44:50.276593+0000","last_became_active":"2024-04-24T23:42:51.072651+0000","last_became_peered":"2024-04-24T23:42:51.072651+0000","last_unstale":"2024-04-24T23:44:50.276593+0000","last_undegraded":"2024-04-24T23:44:50.276593+0000","last_fullsized":"2024-04-24T23:44:50.276593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:59:35.348841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071625+0000","last_change":"2024-04-24T23:42:55.322461+0000","last_active":"2024-04-24T23:44:50.071625+0000","last_peered":"2024-04-24T23:44:50.071625+0000","last_clean":"2024-04-24T23:44:50.071625+0000","last_became_active":"2024-04-24T23:42:55.321685+0000","last_became_peered":"2024-04-24T23:42:55.321685+0000","last_unstale":"2024-04-24T23:44:50.071625+0000","last_undegraded":"2024-04-24T23:44:50.071625+0000","last_fullsized":"2024-04-24T23:44:50.071625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:16:59.495122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074618+0000","last_change":"2024-04-24T23:43:49.235090+0000","last_active":"2024-04-24T23:44:50.074618+0000","last_peered":"2024-04-24T23:44:50.074618+0000","last_clean":"2024-04-24T23:44:50.074618+0000","last_became_active":"2024-04-24T23:43:49.223773+0000","last_became_peered":"2024-04-24T23:43:49.223773+0000","last_unstale":"2024-04-24T23:44:50.074618+0000","last_undegraded":"2024-04-24T23:44:50.074618+0000","last_fullsized":"2024-04-24T23:44:50.074618+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:14:23.776176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074166+0000","last_change":"2024-04-24T23:42:51.084691+0000","last_active":"2024-04-24T23:44:50.074166+0000","last_peered":"2024-04-24T23:44:50.074166+0000","last_clean":"2024-04-24T23:44:50.074166+0000","last_became_active":"2024-04-24T23:42:51.084304+0000","last_became_peered":"2024-04-24T23:42:51.084304+0000","last_unstale":"2024-04-24T23:44:50.074166+0000","last_undegraded":"2024-04-24T23:44:50.074166+0000","last_fullsized":"2024-04-24T23:44:50.074166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:31.524026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071760+0000","last_change":"2024-04-24T23:42:55.323045+0000","last_active":"2024-04-24T23:44:50.071760+0000","last_peered":"2024-04-24T23:44:50.071760+0000","last_clean":"2024-04-24T23:44:50.071760+0000","last_became_active":"2024-04-24T23:42:55.321387+0000","last_became_peered":"2024-04-24T23:42:55.321387+0000","last_unstale":"2024-04-24T23:44:50.071760+0000","last_undegraded":"2024-04-24T23:44:50.071760+0000","last_fullsized":"2024-04-24T23:44:50.071760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:21.064626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073467+0000","last_change":"2024-04-24T23:42:50.068868+0000","last_active":"2024-04-24T23:44:50.073467+0000","last_peered":"2024-04-24T23:44:50.073467+0000","last_clean":"2024-04-24T23:44:50.073467+0000","last_became_active":"2024-04-24T23:42:50.068427+0000","last_became_peered":"2024-04-24T23:42:50.068427+0000","last_unstale":"2024-04-24T23:44:50.073467+0000","last_undegraded":"2024-04-24T23:44:50.073467+0000","last_fullsized":"2024-04-24T23:44:50.073467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:45.344629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071740+0000","last_change":"2024-04-24T23:42:51.080262+0000","last_active":"2024-04-24T23:44:50.071740+0000","last_peered":"2024-04-24T23:44:50.071740+0000","last_clean":"2024-04-24T23:44:50.071740+0000","last_became_active":"2024-04-24T23:42:51.079510+0000","last_became_peered":"2024-04-24T23:42:51.079510+0000","last_unstale":"2024-04-24T23:44:50.071740+0000","last_undegraded":"2024-04-24T23:44:50.071740+0000","last_fullsized":"2024-04-24T23:44:50.071740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:32:06.403691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078499+0000","last_change":"2024-04-24T23:42:55.315772+0000","last_active":"2024-04-24T23:44:50.078499+0000","last_peered":"2024-04-24T23:44:50.078499+0000","last_clean":"2024-04-24T23:44:50.078499+0000","last_became_active":"2024-04-24T23:42:55.315301+0000","last_became_peered":"2024-04-24T23:42:55.315301+0000","last_unstale":"2024-04-24T23:44:50.078499+0000","last_undegraded":"2024-04-24T23:44:50.078499+0000","last_fullsized":"2024-04-24T23:44:50.078499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:49.661520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:16.262165+0000","last_change":"2024-04-24T23:42:50.075356+0000","last_active":"2024-04-24T23:45:16.262165+0000","last_peered":"2024-04-24T23:45:16.262165+0000","last_clean":"2024-04-24T23:45:16.262165+0000","last_became_active":"2024-04-24T23:42:50.074899+0000","last_became_peered":"2024-04-24T23:42:50.074899+0000","last_unstale":"2024-04-24T23:45:16.262165+0000","last_undegraded":"2024-04-24T23:45:16.262165+0000","last_fullsized":"2024-04-24T23:45:16.262165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:20.098652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073965+0000","last_change":"2024-04-24T23:43:49.224709+0000","last_active":"2024-04-24T23:44:50.073965+0000","last_peered":"2024-04-24T23:44:50.073965+0000","last_clean":"2024-04-24T23:44:50.073965+0000","last_became_active":"2024-04-24T23:43:49.224345+0000","last_became_peered":"2024-04-24T23:43:49.224345+0000","last_unstale":"2024-04-24T23:44:50.073965+0000","last_undegraded":"2024-04-24T23:44:50.073965+0000","last_fullsized":"2024-04-24T23:44:50.073965+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:56.093532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078701+0000","last_change":"2024-04-24T23:42:55.321966+0000","last_active":"2024-04-24T23:44:50.078701+0000","last_peered":"2024-04-24T23:44:50.078701+0000","last_clean":"2024-04-24T23:44:50.078701+0000","last_became_active":"2024-04-24T23:42:55.321533+0000","last_became_peered":"2024-04-24T23:42:55.321533+0000","last_unstale":"2024-04-24T23:44:50.078701+0000","last_undegraded":"2024-04-24T23:44:50.078701+0000","last_fullsized":"2024-04-24T23:44:50.078701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:47.410759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226287+0000","last_change":"2024-04-24T23:42:50.067900+0000","last_active":"2024-04-24T23:44:50.226287+0000","last_peered":"2024-04-24T23:44:50.226287+0000","last_clean":"2024-04-24T23:44:50.226287+0000","last_became_active":"2024-04-24T23:42:50.067540+0000","last_became_peered":"2024-04-24T23:42:50.067540+0000","last_unstale":"2024-04-24T23:44:50.226287+0000","last_undegraded":"2024-04-24T23:44:50.226287+0000","last_fullsized":"2024-04-24T23:44:50.226287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:49.367501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071791+0000","last_change":"2024-04-24T23:42:51.072183+0000","last_active":"2024-04-24T23:44:50.071791+0000","last_peered":"2024-04-24T23:44:50.071791+0000","last_clean":"2024-04-24T23:44:50.071791+0000","last_became_active":"2024-04-24T23:42:51.071198+0000","last_became_peered":"2024-04-24T23:42:51.071198+0000","last_unstale":"2024-04-24T23:44:50.071791+0000","last_undegraded":"2024-04-24T23:44:50.071791+0000","last_fullsized":"2024-04-24T23:44:50.071791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:14.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190050+0000","last_change":"2024-04-24T23:42:55.317010+0000","last_active":"2024-04-24T23:44:50.190050+0000","last_peered":"2024-04-24T23:44:50.190050+0000","last_clean":"2024-04-24T23:44:50.190050+0000","last_became_active":"2024-04-24T23:42:55.316602+0000","last_became_peered":"2024-04-24T23:42:55.316602+0000","last_unstale":"2024-04-24T23:44:50.190050+0000","last_undegraded":"2024-04-24T23:44:50.190050+0000","last_fullsized":"2024-04-24T23:44:50.190050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:44.509655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072614+0000","last_change":"2024-04-24T23:42:50.066615+0000","last_active":"2024-04-24T23:44:50.072614+0000","last_peered":"2024-04-24T23:44:50.072614+0000","last_clean":"2024-04-24T23:44:50.072614+0000","last_became_active":"2024-04-24T23:42:50.066172+0000","last_became_peered":"2024-04-24T23:42:50.066172+0000","last_unstale":"2024-04-24T23:44:50.072614+0000","last_undegraded":"2024-04-24T23:44:50.072614+0000","last_fullsized":"2024-04-24T23:44:50.072614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:29.089971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224223+0000","last_change":"2024-04-24T23:43:49.226164+0000","last_active":"2024-04-24T23:44:50.224223+0000","last_peered":"2024-04-24T23:44:50.224223+0000","last_clean":"2024-04-24T23:44:50.224223+0000","last_became_active":"2024-04-24T23:43:49.225806+0000","last_became_peered":"2024-04-24T23:43:49.225806+0000","last_unstale":"2024-04-24T23:44:50.224223+0000","last_undegraded":"2024-04-24T23:44:50.224223+0000","last_fullsized":"2024-04-24T23:44:50.224223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:56.068947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276432+0000","last_change":"2024-04-24T23:42:55.323829+0000","last_active":"2024-04-24T23:44:50.276432+0000","last_peered":"2024-04-24T23:44:50.276432+0000","last_clean":"2024-04-24T23:44:50.276432+0000","last_became_active":"2024-04-24T23:42:55.317832+0000","last_became_peered":"2024-04-24T23:42:55.317832+0000","last_unstale":"2024-04-24T23:44:50.276432+0000","last_undegraded":"2024-04-24T23:44:50.276432+0000","last_fullsized":"2024-04-24T23:44:50.276432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:55.223048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071398+0000","last_change":"2024-04-24T23:42:50.072970+0000","last_active":"2024-04-24T23:44:50.071398+0000","last_peered":"2024-04-24T23:44:50.071398+0000","last_clean":"2024-04-24T23:44:50.071398+0000","last_became_active":"2024-04-24T23:42:50.072645+0000","last_became_peered":"2024-04-24T23:42:50.072645+0000","last_unstale":"2024-04-24T23:44:50.071398+0000","last_undegraded":"2024-04-24T23:44:50.071398+0000","last_fullsized":"2024-04-24T23:44:50.071398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:32.990558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224569+0000","last_change":"2024-04-24T23:42:51.076799+0000","last_active":"2024-04-24T23:44:50.224569+0000","last_peered":"2024-04-24T23:44:50.224569+0000","last_clean":"2024-04-24T23:44:50.224569+0000","last_became_active":"2024-04-24T23:42:51.076461+0000","last_became_peered":"2024-04-24T23:42:51.076461+0000","last_unstale":"2024-04-24T23:44:50.224569+0000","last_undegraded":"2024-04-24T23:44:50.224569+0000","last_fullsized":"2024-04-24T23:44:50.224569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:17.193086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227216+0000","last_change":"2024-04-24T23:42:55.326445+0000","last_active":"2024-04-24T23:44:50.227216+0000","last_peered":"2024-04-24T23:44:50.227216+0000","last_clean":"2024-04-24T23:44:50.227216+0000","last_became_active":"2024-04-24T23:42:55.325620+0000","last_became_peered":"2024-04-24T23:42:55.325620+0000","last_unstale":"2024-04-24T23:44:50.227216+0000","last_undegraded":"2024-04-24T23:44:50.227216+0000","last_fullsized":"2024-04-24T23:44:50.227216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:58:54.403014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.303688+0000","last_change":"2024-04-24T23:42:50.069541+0000","last_active":"2024-04-24T23:45:15.303688+0000","last_peered":"2024-04-24T23:45:15.303688+0000","last_clean":"2024-04-24T23:45:15.303688+0000","last_became_active":"2024-04-24T23:42:50.069210+0000","last_became_peered":"2024-04-24T23:42:50.069210+0000","last_unstale":"2024-04-24T23:45:15.303688+0000","last_undegraded":"2024-04-24T23:45:15.303688+0000","last_fullsized":"2024-04-24T23:45:15.303688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:20.999246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074660+0000","last_change":"2024-04-24T23:42:51.084745+0000","last_active":"2024-04-24T23:44:50.074660+0000","last_peered":"2024-04-24T23:44:50.074660+0000","last_clean":"2024-04-24T23:44:50.074660+0000","last_became_active":"2024-04-24T23:42:51.084396+0000","last_became_peered":"2024-04-24T23:42:51.084396+0000","last_unstale":"2024-04-24T23:44:50.074660+0000","last_undegraded":"2024-04-24T23:44:50.074660+0000","last_fullsized":"2024-04-24T23:44:50.074660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:55:30.969113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224398+0000","last_change":"2024-04-24T23:42:55.319929+0000","last_active":"2024-04-24T23:44:50.224398+0000","last_peered":"2024-04-24T23:44:50.224398+0000","last_clean":"2024-04-24T23:44:50.224398+0000","last_became_active":"2024-04-24T23:42:55.319505+0000","last_became_peered":"2024-04-24T23:42:55.319505+0000","last_unstale":"2024-04-24T23:44:50.224398+0000","last_undegraded":"2024-04-24T23:44:50.224398+0000","last_fullsized":"2024-04-24T23:44:50.224398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:48.864268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228258+0000","last_change":"2024-04-24T23:43:49.226781+0000","last_active":"2024-04-24T23:44:50.228258+0000","last_peered":"2024-04-24T23:44:50.228258+0000","last_clean":"2024-04-24T23:44:50.228258+0000","last_became_active":"2024-04-24T23:43:49.222089+0000","last_became_peered":"2024-04-24T23:43:49.222089+0000","last_unstale":"2024-04-24T23:44:50.228258+0000","last_undegraded":"2024-04-24T23:44:50.228258+0000","last_fullsized":"2024-04-24T23:44:50.228258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:57.224621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276509+0000","last_change":"2024-04-24T23:42:51.074546+0000","last_active":"2024-04-24T23:44:50.276509+0000","last_peered":"2024-04-24T23:44:50.276509+0000","last_clean":"2024-04-24T23:44:50.276509+0000","last_became_active":"2024-04-24T23:42:51.074203+0000","last_became_peered":"2024-04-24T23:42:51.074203+0000","last_unstale":"2024-04-24T23:44:50.276509+0000","last_undegraded":"2024-04-24T23:44:50.276509+0000","last_fullsized":"2024-04-24T23:44:50.276509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:53.727502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071804+0000","last_change":"2024-04-24T23:42:55.322581+0000","last_active":"2024-04-24T23:44:50.071804+0000","last_peered":"2024-04-24T23:44:50.071804+0000","last_clean":"2024-04-24T23:44:50.071804+0000","last_became_active":"2024-04-24T23:42:55.322050+0000","last_became_peered":"2024-04-24T23:42:55.322050+0000","last_unstale":"2024-04-24T23:44:50.071804+0000","last_undegraded":"2024-04-24T23:44:50.071804+0000","last_fullsized":"2024-04-24T23:44:50.071804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:44.493423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071732+0000","last_change":"2024-04-24T23:42:50.071274+0000","last_active":"2024-04-24T23:44:50.071732+0000","last_peered":"2024-04-24T23:44:50.071732+0000","last_clean":"2024-04-24T23:44:50.071732+0000","last_became_active":"2024-04-24T23:42:50.070908+0000","last_became_peered":"2024-04-24T23:42:50.070908+0000","last_unstale":"2024-04-24T23:44:50.071732+0000","last_undegraded":"2024-04-24T23:44:50.071732+0000","last_fullsized":"2024-04-24T23:44:50.071732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:28.826913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074559+0000","last_change":"2024-04-24T23:42:51.071190+0000","last_active":"2024-04-24T23:44:50.074559+0000","last_peered":"2024-04-24T23:44:50.074559+0000","last_clean":"2024-04-24T23:44:50.074559+0000","last_became_active":"2024-04-24T23:42:51.070805+0000","last_became_peered":"2024-04-24T23:42:51.070805+0000","last_unstale":"2024-04-24T23:44:50.074559+0000","last_undegraded":"2024-04-24T23:44:50.074559+0000","last_fullsized":"2024-04-24T23:44:50.074559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:28.664403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224451+0000","last_change":"2024-04-24T23:42:55.319229+0000","last_active":"2024-04-24T23:44:50.224451+0000","last_peered":"2024-04-24T23:44:50.224451+0000","last_clean":"2024-04-24T23:44:50.224451+0000","last_became_active":"2024-04-24T23:42:55.318787+0000","last_became_peered":"2024-04-24T23:42:55.318787+0000","last_unstale":"2024-04-24T23:44:50.224451+0000","last_undegraded":"2024-04-24T23:44:50.224451+0000","last_fullsized":"2024-04-24T23:44:50.224451+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:03.973200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'41","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:15.260922+0000","last_change":"2024-04-24T23:42:50.071564+0000","last_active":"2024-04-24T23:45:15.260922+0000","last_peered":"2024-04-24T23:45:15.260922+0000","last_clean":"2024-04-24T23:45:15.260922+0000","last_became_active":"2024-04-24T23:42:50.071218+0000","last_became_peered":"2024-04-24T23:42:50.071218+0000","last_unstale":"2024-04-24T23:45:15.260922+0000","last_undegraded":"2024-04-24T23:45:15.260922+0000","last_fullsized":"2024-04-24T23:45:15.260922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:26.432150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68263,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":189,"num_write":43,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074503+0000","last_change":"2024-04-24T23:42:51.072067+0000","last_active":"2024-04-24T23:44:50.074503+0000","last_peered":"2024-04-24T23:44:50.074503+0000","last_clean":"2024-04-24T23:44:50.074503+0000","last_became_active":"2024-04-24T23:42:51.070336+0000","last_became_peered":"2024-04-24T23:42:51.070336+0000","last_unstale":"2024-04-24T23:44:50.074503+0000","last_undegraded":"2024-04-24T23:44:50.074503+0000","last_fullsized":"2024-04-24T23:44:50.074503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:13:59.348085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072028+0000","last_change":"2024-04-24T23:42:55.319610+0000","last_active":"2024-04-24T23:44:50.072028+0000","last_peered":"2024-04-24T23:44:50.072028+0000","last_clean":"2024-04-24T23:44:50.072028+0000","last_became_active":"2024-04-24T23:42:55.319173+0000","last_became_peered":"2024-04-24T23:42:55.319173+0000","last_unstale":"2024-04-24T23:44:50.072028+0000","last_undegraded":"2024-04-24T23:44:50.072028+0000","last_fullsized":"2024-04-24T23:44:50.072028+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:08.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078708+0000","last_change":"2024-04-24T23:42:50.071903+0000","last_active":"2024-04-24T23:44:50.078708+0000","last_peered":"2024-04-24T23:44:50.078708+0000","last_clean":"2024-04-24T23:44:50.078708+0000","last_became_active":"2024-04-24T23:42:50.070623+0000","last_became_peered":"2024-04-24T23:42:50.070623+0000","last_unstale":"2024-04-24T23:44:50.078708+0000","last_undegraded":"2024-04-24T23:44:50.078708+0000","last_fullsized":"2024-04-24T23:44:50.078708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:15.842995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072796+0000","last_change":"2024-04-24T23:42:51.069222+0000","last_active":"2024-04-24T23:44:50.072796+0000","last_peered":"2024-04-24T23:44:50.072796+0000","last_clean":"2024-04-24T23:44:50.072796+0000","last_became_active":"2024-04-24T23:42:51.068945+0000","last_became_peered":"2024-04-24T23:42:51.068945+0000","last_unstale":"2024-04-24T23:44:50.072796+0000","last_undegraded":"2024-04-24T23:44:50.072796+0000","last_fullsized":"2024-04-24T23:44:50.072796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:13.621721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189700+0000","last_change":"2024-04-24T23:42:55.314322+0000","last_active":"2024-04-24T23:44:50.189700+0000","last_peered":"2024-04-24T23:44:50.189700+0000","last_clean":"2024-04-24T23:44:50.189700+0000","last_became_active":"2024-04-24T23:42:55.313973+0000","last_became_peered":"2024-04-24T23:42:55.313973+0000","last_unstale":"2024-04-24T23:44:50.189700+0000","last_undegraded":"2024-04-24T23:44:50.189700+0000","last_fullsized":"2024-04-24T23:44:50.189700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:52.711697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071783+0000","last_change":"2024-04-24T23:42:50.076133+0000","last_active":"2024-04-24T23:44:50.071783+0000","last_peered":"2024-04-24T23:44:50.071783+0000","last_clean":"2024-04-24T23:44:50.071783+0000","last_became_active":"2024-04-24T23:42:50.075623+0000","last_became_peered":"2024-04-24T23:42:50.075623+0000","last_unstale":"2024-04-24T23:44:50.071783+0000","last_undegraded":"2024-04-24T23:44:50.071783+0000","last_fullsized":"2024-04-24T23:44:50.071783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:07.013130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228754+0000","last_change":"2024-04-24T23:42:51.071199+0000","last_active":"2024-04-24T23:44:50.228754+0000","last_peered":"2024-04-24T23:44:50.228754+0000","last_clean":"2024-04-24T23:44:50.228754+0000","last_became_active":"2024-04-24T23:42:51.070797+0000","last_became_peered":"2024-04-24T23:42:51.070797+0000","last_unstale":"2024-04-24T23:44:50.228754+0000","last_undegraded":"2024-04-24T23:44:50.228754+0000","last_fullsized":"2024-04-24T23:44:50.228754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:29.829231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224947+0000","last_change":"2024-04-24T23:42:55.323372+0000","last_active":"2024-04-24T23:44:50.224947+0000","last_peered":"2024-04-24T23:44:50.224947+0000","last_clean":"2024-04-24T23:44:50.224947+0000","last_became_active":"2024-04-24T23:42:55.323000+0000","last_became_peered":"2024-04-24T23:42:55.323000+0000","last_unstale":"2024-04-24T23:44:50.224947+0000","last_undegraded":"2024-04-24T23:44:50.224947+0000","last_fullsized":"2024-04-24T23:44:50.224947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:42.960732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.256361+0000","last_change":"2024-04-24T23:42:50.064665+0000","last_active":"2024-04-24T23:45:12.256361+0000","last_peered":"2024-04-24T23:45:12.256361+0000","last_clean":"2024-04-24T23:45:12.256361+0000","last_became_active":"2024-04-24T23:42:50.064276+0000","last_became_peered":"2024-04-24T23:42:50.064276+0000","last_unstale":"2024-04-24T23:45:12.256361+0000","last_undegraded":"2024-04-24T23:45:12.256361+0000","last_fullsized":"2024-04-24T23:45:12.256361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:28.584511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228739+0000","last_change":"2024-04-24T23:42:51.076129+0000","last_active":"2024-04-24T23:44:50.228739+0000","last_peered":"2024-04-24T23:44:50.228739+0000","last_clean":"2024-04-24T23:44:50.228739+0000","last_became_active":"2024-04-24T23:42:51.075556+0000","last_became_peered":"2024-04-24T23:42:51.075556+0000","last_unstale":"2024-04-24T23:44:50.228739+0000","last_undegraded":"2024-04-24T23:44:50.228739+0000","last_fullsized":"2024-04-24T23:44:50.228739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:24.671395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073186+0000","last_change":"2024-04-24T23:42:55.327363+0000","last_active":"2024-04-24T23:44:50.073186+0000","last_peered":"2024-04-24T23:44:50.073186+0000","last_clean":"2024-04-24T23:44:50.073186+0000","last_became_active":"2024-04-24T23:42:55.326967+0000","last_became_peered":"2024-04-24T23:42:55.326967+0000","last_unstale":"2024-04-24T23:44:50.073186+0000","last_undegraded":"2024-04-24T23:44:50.073186+0000","last_fullsized":"2024-04-24T23:44:50.073186+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:30.310087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073794+0000","last_change":"2024-04-24T23:42:50.072056+0000","last_active":"2024-04-24T23:44:50.073794+0000","last_peered":"2024-04-24T23:44:50.073794+0000","last_clean":"2024-04-24T23:44:50.073794+0000","last_became_active":"2024-04-24T23:42:50.071614+0000","last_became_peered":"2024-04-24T23:42:50.071614+0000","last_unstale":"2024-04-24T23:44:50.073794+0000","last_undegraded":"2024-04-24T23:44:50.073794+0000","last_fullsized":"2024-04-24T23:44:50.073794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:37.551427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071742+0000","last_change":"2024-04-24T23:43:49.223137+0000","last_active":"2024-04-24T23:44:50.071742+0000","last_peered":"2024-04-24T23:44:50.071742+0000","last_clean":"2024-04-24T23:44:50.071742+0000","last_became_active":"2024-04-24T23:43:49.222775+0000","last_became_peered":"2024-04-24T23:43:49.222775+0000","last_unstale":"2024-04-24T23:44:50.071742+0000","last_undegraded":"2024-04-24T23:44:50.071742+0000","last_fullsized":"2024-04-24T23:44:50.071742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:22.910320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071553+0000","last_change":"2024-04-24T23:42:55.319364+0000","last_active":"2024-04-24T23:44:50.071553+0000","last_peered":"2024-04-24T23:44:50.071553+0000","last_clean":"2024-04-24T23:44:50.071553+0000","last_became_active":"2024-04-24T23:42:55.318858+0000","last_became_peered":"2024-04-24T23:42:55.318858+0000","last_unstale":"2024-04-24T23:44:50.071553+0000","last_undegraded":"2024-04-24T23:44:50.071553+0000","last_fullsized":"2024-04-24T23:44:50.071553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:00.065796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074870+0000","last_change":"2024-04-24T23:42:50.073806+0000","last_active":"2024-04-24T23:44:50.074870+0000","last_peered":"2024-04-24T23:44:50.074870+0000","last_clean":"2024-04-24T23:44:50.074870+0000","last_became_active":"2024-04-24T23:42:50.072378+0000","last_became_peered":"2024-04-24T23:42:50.072378+0000","last_unstale":"2024-04-24T23:44:50.074870+0000","last_undegraded":"2024-04-24T23:44:50.074870+0000","last_fullsized":"2024-04-24T23:44:50.074870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:05.624277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190122+0000","last_change":"2024-04-24T23:42:51.075538+0000","last_active":"2024-04-24T23:44:50.190122+0000","last_peered":"2024-04-24T23:44:50.190122+0000","last_clean":"2024-04-24T23:44:50.190122+0000","last_became_active":"2024-04-24T23:42:51.072635+0000","last_became_peered":"2024-04-24T23:42:51.072635+0000","last_unstale":"2024-04-24T23:44:50.190122+0000","last_undegraded":"2024-04-24T23:44:50.190122+0000","last_fullsized":"2024-04-24T23:44:50.190122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:14.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073941+0000","last_change":"2024-04-24T23:42:55.315820+0000","last_active":"2024-04-24T23:44:50.073941+0000","last_peered":"2024-04-24T23:44:50.073941+0000","last_clean":"2024-04-24T23:44:50.073941+0000","last_became_active":"2024-04-24T23:42:55.314961+0000","last_became_peered":"2024-04-24T23:42:55.314961+0000","last_unstale":"2024-04-24T23:44:50.073941+0000","last_undegraded":"2024-04-24T23:44:50.073941+0000","last_fullsized":"2024-04-24T23:44:50.073941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:55.436691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.224709+0000","last_change":"2024-04-24T23:42:50.071752+0000","last_active":"2024-04-24T23:44:50.224709+0000","last_peered":"2024-04-24T23:44:50.224709+0000","last_clean":"2024-04-24T23:44:50.224709+0000","last_became_active":"2024-04-24T23:42:50.071448+0000","last_became_peered":"2024-04-24T23:42:50.071448+0000","last_unstale":"2024-04-24T23:44:50.224709+0000","last_undegraded":"2024-04-24T23:44:50.224709+0000","last_fullsized":"2024-04-24T23:44:50.224709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:05.790727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226822+0000","last_change":"2024-04-24T23:42:51.078082+0000","last_active":"2024-04-24T23:44:50.226822+0000","last_peered":"2024-04-24T23:44:50.226822+0000","last_clean":"2024-04-24T23:44:50.226822+0000","last_became_active":"2024-04-24T23:42:51.077709+0000","last_became_peered":"2024-04-24T23:42:51.077709+0000","last_unstale":"2024-04-24T23:44:50.226822+0000","last_undegraded":"2024-04-24T23:44:50.226822+0000","last_fullsized":"2024-04-24T23:44:50.226822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:51.189445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072855+0000","last_change":"2024-04-24T23:42:55.319359+0000","last_active":"2024-04-24T23:44:50.072855+0000","last_peered":"2024-04-24T23:44:50.072855+0000","last_clean":"2024-04-24T23:44:50.072855+0000","last_became_active":"2024-04-24T23:42:55.318890+0000","last_became_peered":"2024-04-24T23:42:55.318890+0000","last_unstale":"2024-04-24T23:44:50.072855+0000","last_undegraded":"2024-04-24T23:44:50.072855+0000","last_fullsized":"2024-04-24T23:44:50.072855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:32.986354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.257199+0000","last_change":"2024-04-24T23:42:50.067676+0000","last_active":"2024-04-24T23:45:13.257199+0000","last_peered":"2024-04-24T23:45:13.257199+0000","last_clean":"2024-04-24T23:45:13.257199+0000","last_became_active":"2024-04-24T23:42:50.067130+0000","last_became_peered":"2024-04-24T23:42:50.067130+0000","last_unstale":"2024-04-24T23:45:13.257199+0000","last_undegraded":"2024-04-24T23:45:13.257199+0000","last_fullsized":"2024-04-24T23:45:13.257199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:16.108034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.226654+0000","last_change":"2024-04-24T23:42:51.079910+0000","last_active":"2024-04-24T23:44:50.226654+0000","last_peered":"2024-04-24T23:44:50.226654+0000","last_clean":"2024-04-24T23:44:50.226654+0000","last_became_active":"2024-04-24T23:42:51.079598+0000","last_became_peered":"2024-04-24T23:42:51.079598+0000","last_unstale":"2024-04-24T23:44:50.226654+0000","last_undegraded":"2024-04-24T23:44:50.226654+0000","last_fullsized":"2024-04-24T23:44:50.226654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:03.743661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073330+0000","last_change":"2024-04-24T23:42:55.321767+0000","last_active":"2024-04-24T23:44:50.073330+0000","last_peered":"2024-04-24T23:44:50.073330+0000","last_clean":"2024-04-24T23:44:50.073330+0000","last_became_active":"2024-04-24T23:42:55.321398+0000","last_became_peered":"2024-04-24T23:42:55.321398+0000","last_unstale":"2024-04-24T23:44:50.073330+0000","last_undegraded":"2024-04-24T23:44:50.073330+0000","last_fullsized":"2024-04-24T23:44:50.073330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:39.773856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071894+0000","last_change":"2024-04-24T23:42:51.078289+0000","last_active":"2024-04-24T23:44:50.071894+0000","last_peered":"2024-04-24T23:44:50.071894+0000","last_clean":"2024-04-24T23:44:50.071894+0000","last_became_active":"2024-04-24T23:42:51.076605+0000","last_became_peered":"2024-04-24T23:42:51.076605+0000","last_unstale":"2024-04-24T23:44:50.071894+0000","last_undegraded":"2024-04-24T23:44:50.071894+0000","last_fullsized":"2024-04-24T23:44:50.071894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:03.314605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":153,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:12.255804+0000","last_change":"2024-04-24T23:42:50.069103+0000","last_active":"2024-04-24T23:45:12.255804+0000","last_peered":"2024-04-24T23:45:12.255804+0000","last_clean":"2024-04-24T23:45:12.255804+0000","last_became_active":"2024-04-24T23:42:50.068792+0000","last_became_peered":"2024-04-24T23:42:50.068792+0000","last_unstale":"2024-04-24T23:45:12.255804+0000","last_undegraded":"2024-04-24T23:45:12.255804+0000","last_fullsized":"2024-04-24T23:45:12.255804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:51.058744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228121+0000","last_change":"2024-04-24T23:42:55.316791+0000","last_active":"2024-04-24T23:44:50.228121+0000","last_peered":"2024-04-24T23:44:50.228121+0000","last_clean":"2024-04-24T23:44:50.228121+0000","last_became_active":"2024-04-24T23:42:55.316288+0000","last_became_peered":"2024-04-24T23:42:55.316288+0000","last_unstale":"2024-04-24T23:44:50.228121+0000","last_undegraded":"2024-04-24T23:44:50.228121+0000","last_fullsized":"2024-04-24T23:44:50.228121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:37.830193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227286+0000","last_change":"2024-04-24T23:42:51.070787+0000","last_active":"2024-04-24T23:44:50.227286+0000","last_peered":"2024-04-24T23:44:50.227286+0000","last_clean":"2024-04-24T23:44:50.227286+0000","last_became_active":"2024-04-24T23:42:51.070472+0000","last_became_peered":"2024-04-24T23:42:51.070472+0000","last_unstale":"2024-04-24T23:44:50.227286+0000","last_undegraded":"2024-04-24T23:44:50.227286+0000","last_fullsized":"2024-04-24T23:44:50.227286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:22.517739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071805+0000","last_change":"2024-04-24T23:42:50.064611+0000","last_active":"2024-04-24T23:44:50.071805+0000","last_peered":"2024-04-24T23:44:50.071805+0000","last_clean":"2024-04-24T23:44:50.071805+0000","last_became_active":"2024-04-24T23:42:50.064038+0000","last_became_peered":"2024-04-24T23:42:50.064038+0000","last_unstale":"2024-04-24T23:44:50.071805+0000","last_undegraded":"2024-04-24T23:44:50.071805+0000","last_fullsized":"2024-04-24T23:44:50.071805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:31.219931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228428+0000","last_change":"2024-04-24T23:41:34.274015+0000","last_active":"2024-04-24T23:44:50.228428+0000","last_peered":"2024-04-24T23:44:50.228428+0000","last_clean":"2024-04-24T23:44:50.228428+0000","last_became_active":"2024-04-24T23:41:34.273564+0000","last_became_peered":"2024-04-24T23:41:34.273564+0000","last_unstale":"2024-04-24T23:44:50.228428+0000","last_undegraded":"2024-04-24T23:44:50.228428+0000","last_fullsized":"2024-04-24T23:44:50.228428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:38:10.215541+0000","last_clean_scrub_stamp":"2024-04-24T23:38:10.215541+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:15:34.700580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072241+0000","last_change":"2024-04-24T23:42:55.320259+0000","last_active":"2024-04-24T23:44:50.072241+0000","last_peered":"2024-04-24T23:44:50.072241+0000","last_clean":"2024-04-24T23:44:50.072241+0000","last_became_active":"2024-04-24T23:42:55.319845+0000","last_became_peered":"2024-04-24T23:42:55.319845+0000","last_unstale":"2024-04-24T23:44:50.072241+0000","last_undegraded":"2024-04-24T23:44:50.072241+0000","last_fullsized":"2024-04-24T23:44:50.072241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:12:46.254524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072552+0000","last_change":"2024-04-24T23:42:51.077694+0000","last_active":"2024-04-24T23:44:50.072552+0000","last_peered":"2024-04-24T23:44:50.072552+0000","last_clean":"2024-04-24T23:44:50.072552+0000","last_became_active":"2024-04-24T23:42:51.077353+0000","last_became_peered":"2024-04-24T23:42:51.077353+0000","last_unstale":"2024-04-24T23:44:50.072552+0000","last_undegraded":"2024-04-24T23:44:50.072552+0000","last_fullsized":"2024-04-24T23:44:50.072552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:44.168735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227689+0000","last_change":"2024-04-24T23:42:50.072161+0000","last_active":"2024-04-24T23:44:50.227689+0000","last_peered":"2024-04-24T23:44:50.227689+0000","last_clean":"2024-04-24T23:44:50.227689+0000","last_became_active":"2024-04-24T23:42:50.071817+0000","last_became_peered":"2024-04-24T23:42:50.071817+0000","last_unstale":"2024-04-24T23:44:50.227689+0000","last_undegraded":"2024-04-24T23:44:50.227689+0000","last_fullsized":"2024-04-24T23:44:50.227689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:11.154078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189830+0000","last_change":"2024-04-24T23:42:55.315201+0000","last_active":"2024-04-24T23:44:50.189830+0000","last_peered":"2024-04-24T23:44:50.189830+0000","last_clean":"2024-04-24T23:44:50.189830+0000","last_became_active":"2024-04-24T23:42:55.314730+0000","last_became_peered":"2024-04-24T23:42:55.314730+0000","last_unstale":"2024-04-24T23:44:50.189830+0000","last_undegraded":"2024-04-24T23:44:50.189830+0000","last_fullsized":"2024-04-24T23:44:50.189830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:39.308854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189864+0000","last_change":"2024-04-24T23:43:49.235755+0000","last_active":"2024-04-24T23:44:50.189864+0000","last_peered":"2024-04-24T23:44:50.189864+0000","last_clean":"2024-04-24T23:44:50.189864+0000","last_became_active":"2024-04-24T23:43:49.230488+0000","last_became_peered":"2024-04-24T23:43:49.230488+0000","last_unstale":"2024-04-24T23:44:50.189864+0000","last_undegraded":"2024-04-24T23:44:50.189864+0000","last_fullsized":"2024-04-24T23:44:50.189864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:12.034199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073003+0000","last_change":"2024-04-24T23:42:51.077633+0000","last_active":"2024-04-24T23:44:50.073003+0000","last_peered":"2024-04-24T23:44:50.073003+0000","last_clean":"2024-04-24T23:44:50.073003+0000","last_became_active":"2024-04-24T23:42:51.077300+0000","last_became_peered":"2024-04-24T23:42:51.077300+0000","last_unstale":"2024-04-24T23:44:50.073003+0000","last_undegraded":"2024-04-24T23:44:50.073003+0000","last_fullsized":"2024-04-24T23:44:50.073003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:09:54.692381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227292+0000","last_change":"2024-04-24T23:42:55.317928+0000","last_active":"2024-04-24T23:44:50.227292+0000","last_peered":"2024-04-24T23:44:50.227292+0000","last_clean":"2024-04-24T23:44:50.227292+0000","last_became_active":"2024-04-24T23:42:55.317558+0000","last_became_peered":"2024-04-24T23:42:55.317558+0000","last_unstale":"2024-04-24T23:44:50.227292+0000","last_undegraded":"2024-04-24T23:44:50.227292+0000","last_fullsized":"2024-04-24T23:44:50.227292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:20.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:01.190467+0000","last_change":"2024-04-24T23:43:49.236439+0000","last_active":"2024-04-24T23:45:01.190467+0000","last_peered":"2024-04-24T23:45:01.190467+0000","last_clean":"2024-04-24T23:45:01.190467+0000","last_became_active":"2024-04-24T23:43:49.227756+0000","last_became_peered":"2024-04-24T23:43:49.227756+0000","last_unstale":"2024-04-24T23:45:01.190467+0000","last_undegraded":"2024-04-24T23:45:01.190467+0000","last_fullsized":"2024-04-24T23:45:01.190467+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:34.790165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227799+0000","last_change":"2024-04-24T23:42:51.073314+0000","last_active":"2024-04-24T23:44:50.227799+0000","last_peered":"2024-04-24T23:44:50.227799+0000","last_clean":"2024-04-24T23:44:50.227799+0000","last_became_active":"2024-04-24T23:42:51.072461+0000","last_became_peered":"2024-04-24T23:42:51.072461+0000","last_unstale":"2024-04-24T23:44:50.227799+0000","last_undegraded":"2024-04-24T23:44:50.227799+0000","last_fullsized":"2024-04-24T23:44:50.227799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:14:28.913364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078047+0000","last_change":"2024-04-24T23:42:55.317795+0000","last_active":"2024-04-24T23:44:50.078047+0000","last_peered":"2024-04-24T23:44:50.078047+0000","last_clean":"2024-04-24T23:44:50.078047+0000","last_became_active":"2024-04-24T23:42:55.317357+0000","last_became_peered":"2024-04-24T23:42:55.317357+0000","last_unstale":"2024-04-24T23:44:50.078047+0000","last_undegraded":"2024-04-24T23:44:50.078047+0000","last_fullsized":"2024-04-24T23:44:50.078047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:00.601692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071411+0000","last_change":"2024-04-24T23:42:50.068776+0000","last_active":"2024-04-24T23:44:50.071411+0000","last_peered":"2024-04-24T23:44:50.071411+0000","last_clean":"2024-04-24T23:44:50.071411+0000","last_became_active":"2024-04-24T23:42:50.068370+0000","last_became_peered":"2024-04-24T23:42:50.068370+0000","last_unstale":"2024-04-24T23:44:50.071411+0000","last_undegraded":"2024-04-24T23:44:50.071411+0000","last_fullsized":"2024-04-24T23:44:50.071411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:18.436987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276642+0000","last_change":"2024-04-24T23:42:51.071400+0000","last_active":"2024-04-24T23:44:50.276642+0000","last_peered":"2024-04-24T23:44:50.276642+0000","last_clean":"2024-04-24T23:44:50.276642+0000","last_became_active":"2024-04-24T23:42:51.071096+0000","last_became_peered":"2024-04-24T23:42:51.071096+0000","last_unstale":"2024-04-24T23:44:50.276642+0000","last_undegraded":"2024-04-24T23:44:50.276642+0000","last_fullsized":"2024-04-24T23:44:50.276642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:49:08.480914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228660+0000","last_change":"2024-04-24T23:44:50.228080+0000","last_active":"2024-04-24T23:44:50.228660+0000","last_peered":"2024-04-24T23:44:50.228660+0000","last_clean":"2024-04-24T23:44:50.228660+0000","last_became_active":"2024-04-24T23:44:50.227599+0000","last_became_peered":"2024-04-24T23:44:50.227599+0000","last_unstale":"2024-04-24T23:44:50.228660+0000","last_undegraded":"2024-04-24T23:44:50.228660+0000","last_fullsized":"2024-04-24T23:44:50.228660+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:49.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074277+0000","last_change":"2024-04-24T23:42:50.073170+0000","last_active":"2024-04-24T23:44:50.074277+0000","last_peered":"2024-04-24T23:44:50.074277+0000","last_clean":"2024-04-24T23:44:50.074277+0000","last_became_active":"2024-04-24T23:42:50.072308+0000","last_became_peered":"2024-04-24T23:42:50.072308+0000","last_unstale":"2024-04-24T23:44:50.074277+0000","last_undegraded":"2024-04-24T23:44:50.074277+0000","last_fullsized":"2024-04-24T23:44:50.074277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:37.073579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073609+0000","last_change":"2024-04-24T23:42:51.078414+0000","last_active":"2024-04-24T23:44:50.073609+0000","last_peered":"2024-04-24T23:44:50.073609+0000","last_clean":"2024-04-24T23:44:50.073609+0000","last_became_active":"2024-04-24T23:42:51.078074+0000","last_became_peered":"2024-04-24T23:42:51.078074+0000","last_unstale":"2024-04-24T23:44:50.073609+0000","last_undegraded":"2024-04-24T23:44:50.073609+0000","last_fullsized":"2024-04-24T23:44:50.073609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:37.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.229292+0000","last_change":"2024-04-24T23:44:50.229292+0000","last_active":"2024-04-24T23:44:50.229292+0000","last_peered":"2024-04-24T23:44:50.229292+0000","last_clean":"2024-04-24T23:44:50.229292+0000","last_became_active":"2024-04-24T23:44:50.228528+0000","last_became_peered":"2024-04-24T23:44:50.228528+0000","last_unstale":"2024-04-24T23:44:50.229292+0000","last_undegraded":"2024-04-24T23:44:50.229292+0000","last_fullsized":"2024-04-24T23:44:50.229292+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:53.700501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227345+0000","last_change":"2024-04-24T23:42:50.076642+0000","last_active":"2024-04-24T23:44:50.227345+0000","last_peered":"2024-04-24T23:44:50.227345+0000","last_clean":"2024-04-24T23:44:50.227345+0000","last_became_active":"2024-04-24T23:42:50.076245+0000","last_became_peered":"2024-04-24T23:42:50.076245+0000","last_unstale":"2024-04-24T23:44:50.227345+0000","last_undegraded":"2024-04-24T23:44:50.227345+0000","last_fullsized":"2024-04-24T23:44:50.227345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:38.091054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189662+0000","last_change":"2024-04-24T23:42:51.077272+0000","last_active":"2024-04-24T23:44:50.189662+0000","last_peered":"2024-04-24T23:44:50.189662+0000","last_clean":"2024-04-24T23:44:50.189662+0000","last_became_active":"2024-04-24T23:42:51.076876+0000","last_became_peered":"2024-04-24T23:42:51.076876+0000","last_unstale":"2024-04-24T23:44:50.189662+0000","last_undegraded":"2024-04-24T23:44:50.189662+0000","last_fullsized":"2024-04-24T23:44:50.189662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:07.877623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071954+0000","last_change":"2024-04-24T23:42:55.323795+0000","last_active":"2024-04-24T23:44:50.071954+0000","last_peered":"2024-04-24T23:44:50.071954+0000","last_clean":"2024-04-24T23:44:50.071954+0000","last_became_active":"2024-04-24T23:42:55.322775+0000","last_became_peered":"2024-04-24T23:42:55.322775+0000","last_unstale":"2024-04-24T23:44:50.071954+0000","last_undegraded":"2024-04-24T23:44:50.071954+0000","last_fullsized":"2024-04-24T23:44:50.071954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:41.883150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.077868+0000","last_change":"2024-04-24T23:42:50.071714+0000","last_active":"2024-04-24T23:44:50.077868+0000","last_peered":"2024-04-24T23:44:50.077868+0000","last_clean":"2024-04-24T23:44:50.077868+0000","last_became_active":"2024-04-24T23:42:50.070467+0000","last_became_peered":"2024-04-24T23:42:50.070467+0000","last_unstale":"2024-04-24T23:44:50.077868+0000","last_undegraded":"2024-04-24T23:44:50.077868+0000","last_fullsized":"2024-04-24T23:44:50.077868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:03.521528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227346+0000","last_change":"2024-04-24T23:42:51.080502+0000","last_active":"2024-04-24T23:44:50.227346+0000","last_peered":"2024-04-24T23:44:50.227346+0000","last_clean":"2024-04-24T23:44:50.227346+0000","last_became_active":"2024-04-24T23:42:51.080005+0000","last_became_peered":"2024-04-24T23:42:51.080005+0000","last_unstale":"2024-04-24T23:44:50.227346+0000","last_undegraded":"2024-04-24T23:44:50.227346+0000","last_fullsized":"2024-04-24T23:44:50.227346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.212147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072175+0000","last_change":"2024-04-24T23:42:55.323572+0000","last_active":"2024-04-24T23:44:50.072175+0000","last_peered":"2024-04-24T23:44:50.072175+0000","last_clean":"2024-04-24T23:44:50.072175+0000","last_became_active":"2024-04-24T23:42:55.323143+0000","last_became_peered":"2024-04-24T23:42:55.323143+0000","last_unstale":"2024-04-24T23:44:50.072175+0000","last_undegraded":"2024-04-24T23:44:50.072175+0000","last_fullsized":"2024-04-24T23:44:50.072175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:40.546482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.074163+0000","last_change":"2024-04-24T23:42:50.066006+0000","last_active":"2024-04-24T23:44:50.074163+0000","last_peered":"2024-04-24T23:44:50.074163+0000","last_clean":"2024-04-24T23:44:50.074163+0000","last_became_active":"2024-04-24T23:42:50.065627+0000","last_became_peered":"2024-04-24T23:42:50.065627+0000","last_unstale":"2024-04-24T23:44:50.074163+0000","last_undegraded":"2024-04-24T23:44:50.074163+0000","last_fullsized":"2024-04-24T23:44:50.074163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:43.020687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228420+0000","last_change":"2024-04-24T23:42:51.073063+0000","last_active":"2024-04-24T23:44:50.228420+0000","last_peered":"2024-04-24T23:44:50.228420+0000","last_clean":"2024-04-24T23:44:50.228420+0000","last_became_active":"2024-04-24T23:42:51.072614+0000","last_became_peered":"2024-04-24T23:42:51.072614+0000","last_unstale":"2024-04-24T23:44:50.228420+0000","last_undegraded":"2024-04-24T23:44:50.228420+0000","last_fullsized":"2024-04-24T23:44:50.228420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:58:52.040317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.230519+0000","last_change":"2024-04-24T23:44:50.230008+0000","last_active":"2024-04-24T23:44:50.230519+0000","last_peered":"2024-04-24T23:44:50.230519+0000","last_clean":"2024-04-24T23:44:50.230519+0000","last_became_active":"2024-04-24T23:44:50.229452+0000","last_became_peered":"2024-04-24T23:44:50.229452+0000","last_unstale":"2024-04-24T23:44:50.230519+0000","last_undegraded":"2024-04-24T23:44:50.230519+0000","last_fullsized":"2024-04-24T23:44:50.230519+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:13.691069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:45:13.300487+0000","last_change":"2024-04-24T23:42:50.068237+0000","last_active":"2024-04-24T23:45:13.300487+0000","last_peered":"2024-04-24T23:45:13.300487+0000","last_clean":"2024-04-24T23:45:13.300487+0000","last_became_active":"2024-04-24T23:42:50.067380+0000","last_became_peered":"2024-04-24T23:42:50.067380+0000","last_unstale":"2024-04-24T23:45:13.300487+0000","last_undegraded":"2024-04-24T23:45:13.300487+0000","last_fullsized":"2024-04-24T23:45:13.300487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:35:28.414569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1478,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228299+0000","last_change":"2024-04-24T23:42:51.080037+0000","last_active":"2024-04-24T23:44:50.228299+0000","last_peered":"2024-04-24T23:44:50.228299+0000","last_clean":"2024-04-24T23:44:50.228299+0000","last_became_active":"2024-04-24T23:42:51.079639+0000","last_became_peered":"2024-04-24T23:42:51.079639+0000","last_unstale":"2024-04-24T23:44:50.228299+0000","last_undegraded":"2024-04-24T23:44:50.228299+0000","last_fullsized":"2024-04-24T23:44:50.228299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:08.334755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276473+0000","last_change":"2024-04-24T23:42:55.326539+0000","last_active":"2024-04-24T23:44:50.276473+0000","last_peered":"2024-04-24T23:44:50.276473+0000","last_clean":"2024-04-24T23:44:50.276473+0000","last_became_active":"2024-04-24T23:42:55.326133+0000","last_became_peered":"2024-04-24T23:42:55.326133+0000","last_unstale":"2024-04-24T23:44:50.276473+0000","last_undegraded":"2024-04-24T23:44:50.276473+0000","last_fullsized":"2024-04-24T23:44:50.276473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:30:32.635708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071951+0000","last_change":"2024-04-24T23:42:50.076194+0000","last_active":"2024-04-24T23:44:50.071951+0000","last_peered":"2024-04-24T23:44:50.071951+0000","last_clean":"2024-04-24T23:44:50.071951+0000","last_became_active":"2024-04-24T23:42:50.075717+0000","last_became_peered":"2024-04-24T23:42:50.075717+0000","last_unstale":"2024-04-24T23:44:50.071951+0000","last_undegraded":"2024-04-24T23:44:50.071951+0000","last_fullsized":"2024-04-24T23:44:50.071951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:50:36.971512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228018+0000","last_change":"2024-04-24T23:42:51.072975+0000","last_active":"2024-04-24T23:44:50.228018+0000","last_peered":"2024-04-24T23:44:50.228018+0000","last_clean":"2024-04-24T23:44:50.228018+0000","last_became_active":"2024-04-24T23:42:51.072555+0000","last_became_peered":"2024-04-24T23:42:51.072555+0000","last_unstale":"2024-04-24T23:44:50.228018+0000","last_undegraded":"2024-04-24T23:44:50.228018+0000","last_fullsized":"2024-04-24T23:44:50.228018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:29.786705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071162+0000","last_change":"2024-04-24T23:42:55.316428+0000","last_active":"2024-04-24T23:44:50.071162+0000","last_peered":"2024-04-24T23:44:50.071162+0000","last_clean":"2024-04-24T23:44:50.071162+0000","last_became_active":"2024-04-24T23:42:55.316019+0000","last_became_peered":"2024-04-24T23:42:55.316019+0000","last_unstale":"2024-04-24T23:44:50.071162+0000","last_undegraded":"2024-04-24T23:44:50.071162+0000","last_fullsized":"2024-04-24T23:44:50.071162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:54.292008+0000","last_clean_scrub_stamp":"2024-04-24T23:42:54.292008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:25.300341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073134+0000","last_change":"2024-04-24T23:42:50.074139+0000","last_active":"2024-04-24T23:44:50.073134+0000","last_peered":"2024-04-24T23:44:50.073134+0000","last_clean":"2024-04-24T23:44:50.073134+0000","last_became_active":"2024-04-24T23:42:50.073732+0000","last_became_peered":"2024-04-24T23:42:50.073732+0000","last_unstale":"2024-04-24T23:44:50.073134+0000","last_undegraded":"2024-04-24T23:44:50.073134+0000","last_fullsized":"2024-04-24T23:44:50.073134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:49.039121+0000","last_clean_scrub_stamp":"2024-04-24T23:42:49.039121+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:09.179796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.228522+0000","last_change":"2024-04-24T23:42:51.071369+0000","last_active":"2024-04-24T23:44:50.228522+0000","last_peered":"2024-04-24T23:44:50.228522+0000","last_clean":"2024-04-24T23:44:50.228522+0000","last_became_active":"2024-04-24T23:42:51.071040+0000","last_became_peered":"2024-04-24T23:42:51.071040+0000","last_unstale":"2024-04-24T23:44:50.228522+0000","last_undegraded":"2024-04-24T23:44:50.228522+0000","last_fullsized":"2024-04-24T23:44:50.228522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:53.529562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189686+0000","last_change":"2024-04-24T23:42:51.077367+0000","last_active":"2024-04-24T23:44:50.189686+0000","last_peered":"2024-04-24T23:44:50.189686+0000","last_clean":"2024-04-24T23:44:50.189686+0000","last_became_active":"2024-04-24T23:42:51.076556+0000","last_became_peered":"2024-04-24T23:42:51.076556+0000","last_unstale":"2024-04-24T23:44:50.189686+0000","last_undegraded":"2024-04-24T23:44:50.189686+0000","last_fullsized":"2024-04-24T23:44:50.189686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:01.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078449+0000","last_change":"2024-04-24T23:43:49.223272+0000","last_active":"2024-04-24T23:44:50.078449+0000","last_peered":"2024-04-24T23:44:50.078449+0000","last_clean":"2024-04-24T23:44:50.078449+0000","last_became_active":"2024-04-24T23:43:49.222921+0000","last_became_peered":"2024-04-24T23:43:49.222921+0000","last_unstale":"2024-04-24T23:44:50.078449+0000","last_undegraded":"2024-04-24T23:44:50.078449+0000","last_fullsized":"2024-04-24T23:44:50.078449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:10.443359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.078022+0000","last_change":"2024-04-24T23:42:51.074747+0000","last_active":"2024-04-24T23:44:50.078022+0000","last_peered":"2024-04-24T23:44:50.078022+0000","last_clean":"2024-04-24T23:44:50.078022+0000","last_became_active":"2024-04-24T23:42:51.074334+0000","last_became_peered":"2024-04-24T23:42:51.074334+0000","last_unstale":"2024-04-24T23:44:50.078022+0000","last_undegraded":"2024-04-24T23:44:50.078022+0000","last_fullsized":"2024-04-24T23:44:50.078022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:41:45.825515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.276474+0000","last_change":"2024-04-24T23:42:51.072510+0000","last_active":"2024-04-24T23:44:50.276474+0000","last_peered":"2024-04-24T23:44:50.276474+0000","last_clean":"2024-04-24T23:44:50.276474+0000","last_became_active":"2024-04-24T23:42:51.072159+0000","last_became_peered":"2024-04-24T23:42:51.072159+0000","last_unstale":"2024-04-24T23:44:50.276474+0000","last_undegraded":"2024-04-24T23:44:50.276474+0000","last_fullsized":"2024-04-24T23:44:50.276474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:57:46.655744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.072969+0000","last_change":"2024-04-24T23:43:49.230897+0000","last_active":"2024-04-24T23:44:50.072969+0000","last_peered":"2024-04-24T23:44:50.072969+0000","last_clean":"2024-04-24T23:44:50.072969+0000","last_became_active":"2024-04-24T23:43:49.230514+0000","last_became_peered":"2024-04-24T23:43:49.230514+0000","last_unstale":"2024-04-24T23:44:50.072969+0000","last_undegraded":"2024-04-24T23:44:50.072969+0000","last_fullsized":"2024-04-24T23:44:50.072969+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:24.826152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.190172+0000","last_change":"2024-04-24T23:42:51.073027+0000","last_active":"2024-04-24T23:44:50.190172+0000","last_peered":"2024-04-24T23:44:50.190172+0000","last_clean":"2024-04-24T23:44:50.190172+0000","last_became_active":"2024-04-24T23:42:51.072609+0000","last_became_peered":"2024-04-24T23:42:51.072609+0000","last_unstale":"2024-04-24T23:44:50.190172+0000","last_undegraded":"2024-04-24T23:44:50.190172+0000","last_fullsized":"2024-04-24T23:44:50.190172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:27.159692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.079053+0000","last_change":"2024-04-24T23:42:51.081226+0000","last_active":"2024-04-24T23:44:50.079053+0000","last_peered":"2024-04-24T23:44:50.079053+0000","last_clean":"2024-04-24T23:44:50.079053+0000","last_became_active":"2024-04-24T23:42:51.080821+0000","last_became_peered":"2024-04-24T23:42:51.080821+0000","last_unstale":"2024-04-24T23:44:50.079053+0000","last_undegraded":"2024-04-24T23:44:50.079053+0000","last_fullsized":"2024-04-24T23:44:50.079053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:52.099175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227836+0000","last_change":"2024-04-24T23:42:51.078355+0000","last_active":"2024-04-24T23:44:50.227836+0000","last_peered":"2024-04-24T23:44:50.227836+0000","last_clean":"2024-04-24T23:44:50.227836+0000","last_became_active":"2024-04-24T23:42:51.077056+0000","last_became_peered":"2024-04-24T23:42:51.077056+0000","last_unstale":"2024-04-24T23:44:50.227836+0000","last_undegraded":"2024-04-24T23:44:50.227836+0000","last_fullsized":"2024-04-24T23:44:50.227836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:02:56.602011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.071656+0000","last_change":"2024-04-24T23:42:51.078792+0000","last_active":"2024-04-24T23:44:50.071656+0000","last_peered":"2024-04-24T23:44:50.071656+0000","last_clean":"2024-04-24T23:44:50.071656+0000","last_became_active":"2024-04-24T23:42:51.078453+0000","last_became_peered":"2024-04-24T23:42:51.078453+0000","last_unstale":"2024-04-24T23:44:50.071656+0000","last_undegraded":"2024-04-24T23:44:50.071656+0000","last_fullsized":"2024-04-24T23:44:50.071656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:39.694635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.227382+0000","last_change":"2024-04-24T23:42:51.077638+0000","last_active":"2024-04-24T23:44:50.227382+0000","last_peered":"2024-04-24T23:44:50.227382+0000","last_clean":"2024-04-24T23:44:50.227382+0000","last_became_active":"2024-04-24T23:42:51.077329+0000","last_became_peered":"2024-04-24T23:42:51.077329+0000","last_unstale":"2024-04-24T23:44:50.227382+0000","last_undegraded":"2024-04-24T23:44:50.227382+0000","last_fullsized":"2024-04-24T23:44:50.227382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:25.746543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.075088+0000","last_change":"2024-04-24T23:42:51.073134+0000","last_active":"2024-04-24T23:44:50.075088+0000","last_peered":"2024-04-24T23:44:50.075088+0000","last_clean":"2024-04-24T23:44:50.075088+0000","last_became_active":"2024-04-24T23:42:51.072494+0000","last_became_peered":"2024-04-24T23:42:51.072494+0000","last_unstale":"2024-04-24T23:44:50.075088+0000","last_undegraded":"2024-04-24T23:44:50.075088+0000","last_fullsized":"2024-04-24T23:44:50.075088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:03.478326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.073092+0000","last_change":"2024-04-24T23:42:51.069156+0000","last_active":"2024-04-24T23:44:50.073092+0000","last_peered":"2024-04-24T23:44:50.073092+0000","last_clean":"2024-04-24T23:44:50.073092+0000","last_became_active":"2024-04-24T23:42:51.068834+0000","last_became_peered":"2024-04-24T23:42:51.068834+0000","last_unstale":"2024-04-24T23:44:50.073092+0000","last_undegraded":"2024-04-24T23:44:50.073092+0000","last_fullsized":"2024-04-24T23:44:50.073092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:29:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-24T23:44:50.189908+0000","last_change":"2024-04-24T23:42:51.071653+0000","last_active":"2024-04-24T23:44:50.189908+0000","last_peered":"2024-04-24T23:44:50.189908+0000","last_clean":"2024-04-24T23:44:50.189908+0000","last_became_active":"2024-04-24T23:42:51.071322+0000","last_became_peered":"2024-04-24T23:42:51.071322+0000","last_unstale":"2024-04-24T23:44:50.189908+0000","last_undegraded":"2024-04-24T23:44:50.189908+0000","last_fullsized":"2024-04-24T23:44:50.189908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:42:50.046555+0000","last_clean_scrub_stamp":"2024-04-24T23:42:50.046555+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:58.284540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250026,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75525,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1205,"num_read_kb":685,"num_write":255,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":112,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":221202,"data_compressed":4866,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":520817,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1176008,"data_compressed":64433,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":524423,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":520593,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":1117343,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515831,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":585124,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517595,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":580279,"data_compressed":28247,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":515539,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520437,"data_compressed":26625,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1112393,"data_compressed":62811,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36186,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":70015,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65170,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70531,"data_compressed":1622,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:45:19.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 -- 172.21.15.107:0/2141810105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724107460 msgr2=0x7fc7241098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 --2- 172.21.15.107:0/2141810105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724107460 0x7fc7241098b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009ca0 tx=0x7fc71c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 -- 172.21.15.107:0/2141810105 shutdown_connections 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 --2- 172.21.15.107:0/2141810105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724107460 0x7fc7241098b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 --2- 172.21.15.107:0/2141810105 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724104b30 0x7fc724106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 --2- 172.21.15.107:0/2141810105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 0x7fc7241045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 -- 172.21.15.107:0/2141810105 >> 172.21.15.107:0/2141810105 conn(0x7fc7240fbbf0 msgr2=0x7fc7240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 -- 172.21.15.107:0/2141810105 shutdown_connections 2024-04-24T23:45:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.720+0000 7fc72c37e640 1 -- 172.21.15.107:0/2141810105 wait complete. 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 Processor -- start 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 -- start start 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 0x7fc724101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724107460 0x7fc7241006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc72410d420 con 0x7fc724104b30 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc72410d2a0 con 0x7fc724102200 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72c37e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc72410d5a0 con 0x7fc724107460 2024-04-24T23:45:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc729919640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc729919640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42994/0 (socket says 172.21.15.107:42994) 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc729919640 1 -- 172.21.15.107:0/434407996 learned_addr learned my addr 172.21.15.107:0/434407996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72a11a640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 0x7fc724101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.721+0000 7fc72a91b640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724107460 0x7fc7241006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 -- 172.21.15.107:0/434407996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724107460 msgr2=0x7fc7241006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724107460 0x7fc7241006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 -- 172.21.15.107:0/434407996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 msgr2=0x7fc724101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 0x7fc724101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 -- 172.21.15.107:0/434407996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc724100f80 con 0x7fc724104b30 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc729919640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e9f0 tx=0x7fc71400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71400ccc0 con 0x7fc724104b30 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7241a1b60 con 0x7fc724104b30 2024-04-24T23:45:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc714005500 con 0x7fc724104b30 2024-04-24T23:45:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714010690 con 0x7fc724104b30 2024-04-24T23:45:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.722+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7241a20b0 con 0x7fc724104b30 2024-04-24T23:45:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.723+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7240fd6e0 con 0x7fc724104b30 2024-04-24T23:45:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.724+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc7140046c0 con 0x7fc724104b30 2024-04-24T23:45:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.725+0000 7fc71b7fe640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 0x7fc7100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.725+0000 7fc72a11a640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 0x7fc7100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.725+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc7140cf7c0 con 0x7fc724104b30 2024-04-24T23:45:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.725+0000 7fc72a11a640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 0x7fc7100a5050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc720009bd0 tx=0x7fc720005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:19.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.729+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc71409c350 con 0x7fc724104b30 2024-04-24T23:45:19.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.944+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc724062340 con 0x7fc724104b30 2024-04-24T23:45:19.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.947+0000 7fc71b7fe640 1 -- 172.21.15.107:0/434407996 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7fc71409bc50 con 0x7fc724104b30 2024-04-24T23:45:19.947 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:19.947 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 msgr2=0x7fc7100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 0x7fc7100a5050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc720009bd0 tx=0x7fc720005f90 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 msgr2=0x7fc7241001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e9f0 tx=0x7fc71400eeb0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 shutdown_connections 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc7100a2b90 0x7fc7100a5050 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc724107460 0x7fc7241006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc724104b30 0x7fc7241001b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 --2- 172.21.15.107:0/434407996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc724102200 0x7fc724101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 >> 172.21.15.107:0/434407996 conn(0x7fc7240fbbf0 msgr2=0x7fc724102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 shutdown_connections 2024-04-24T23:45:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:19.949+0000 7fc72c37e640 1 -- 172.21.15.107:0/434407996 wait complete. 2024-04-24T23:45:19.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:45:19.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:45:20.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.074+0000 7f8492334640 1 -- 172.21.15.107:0/2270864153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c101790 msgr2=0x7f848c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.074+0000 7f8492334640 1 --2- 172.21.15.107:0/2270864153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c101790 0x7f848c101bb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f847c009900 tx=0x7f847c02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- 172.21.15.107:0/2270864153 shutdown_connections 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- 172.21.15.107:0/2270864153 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c103340 0x7f848c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- 172.21.15.107:0/2270864153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- 172.21.15.107:0/2270864153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c101790 0x7f848c101bb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- 172.21.15.107:0/2270864153 >> 172.21.15.107:0/2270864153 conn(0x7f848c0fcd20 msgr2=0x7f848c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- 172.21.15.107:0/2270864153 shutdown_connections 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- 172.21.15.107:0/2270864153 wait complete. 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 Processor -- start 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- start start 2024-04-24T23:45:20.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c101790 0x7f848c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10d430 con 0x7f848c103340 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10d2b0 con 0x7f848c102980 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.075+0000 7f8492334640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10d5b0 con 0x7f848c101790 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43000/0 (socket says 172.21.15.107:43000) 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 -- 172.21.15.107:0/929136741 learned_addr learned my addr 172.21.15.107:0/929136741 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f848bfff640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c101790 0x7f848c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f848b7fe640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 -- 172.21.15.107:0/929136741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c101790 msgr2=0x7f848c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c101790 0x7f848c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 -- 172.21.15.107:0/929136741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 msgr2=0x7f848c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 -- 172.21.15.107:0/929136741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c074900 con 0x7f848c103340 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f848b7fe640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84908d1640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8484009fd0 tx=0x7f848400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484017070 con 0x7f848c103340 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848c1a1c50 con 0x7f848c103340 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8484003e50 con 0x7f848c103340 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840106c0 con 0x7f848c103340 2024-04-24T23:45:20.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.076+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f848c1a21a0 con 0x7f848c103340 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.077+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f848c108b20 con 0x7f848c103340 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.078+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8484003970 con 0x7f848c103340 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.079+0000 7f84897fa640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.079+0000 7f848bfff640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.079+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f84840cf120 con 0x7f848c103340 2024-04-24T23:45:20.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.080+0000 7f848bfff640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 0x7f84740a5050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f847c02aa00 tx=0x7f847c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.083+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f848409bcb0 con 0x7f848c103340 2024-04-24T23:45:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.271+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f848c062340 con 0x7f848c103340 2024-04-24T23:45:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.272+0000 7f84897fa640 1 -- 172.21.15.107:0/929136741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f848409b5b0 con 0x7f848c103340 2024-04-24T23:45:20.271 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:45:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 msgr2=0x7f84740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 0x7f84740a5050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f847c02aa00 tx=0x7f847c0315b0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 msgr2=0x7f848c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8484009fd0 tx=0x7f848400b260 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 shutdown_connections 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f848c103340 0x7f848c074070 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c102980 0x7f848c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 --2- 172.21.15.107:0/929136741 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f848c101790 0x7f848c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 >> 172.21.15.107:0/929136741 conn(0x7f848c0fcd20 msgr2=0x7f848c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 shutdown_connections 2024-04-24T23:45:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.274+0000 7f8492334640 1 -- 172.21.15.107:0/929136741 wait complete. 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/434407996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:20 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 -- 172.21.15.107:0/4181381379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c1001b0 msgr2=0x7f8b4c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/4181381379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c109d60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009ca0 tx=0x7f8b4402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 -- 172.21.15.107:0/4181381379 shutdown_connections 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/4181381379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c109d60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/4181381379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c0ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/4181381379 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 0x7f8b4c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 -- 172.21.15.107:0/4181381379 >> 172.21.15.107:0/4181381379 conn(0x7f8b4c0fb370 msgr2=0x7f8b4c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 -- 172.21.15.107:0/4181381379 shutdown_connections 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.388+0000 7f8b52fe9640 1 -- 172.21.15.107:0/4181381379 wait complete. 2024-04-24T23:45:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 Processor -- start 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 -- start start 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c10d400 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c10d280 con 0x7f8b4c1001b0 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b52fe9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c10d580 con 0x7f8b4c102d80 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43024/0 (socket says 172.21.15.107:43024) 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 -- 172.21.15.107:0/554512098 learned_addr learned my addr 172.21.15.107:0/554512098 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b4bfff640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b51586640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 -- 172.21.15.107:0/554512098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 msgr2=0x7f8b4c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 -- 172.21.15.107:0/554512098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 msgr2=0x7f8b4c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b50d85640 1 -- 172.21.15.107:0/554512098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c19f6d0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.389+0000 7f8b4bfff640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b50d85640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000e9c0 tx=0x7f8b4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000cd80 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b400035f0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4c19f9c0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40010690 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.390+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4c19fec0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.391+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4c108be0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.392+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8b400037a0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.393+0000 7f8b49ffb640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 0x7f8b3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.393+0000 7f8b4bfff640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 0x7f8b3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.393+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8b400cf760 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.393+0000 7f8b4bfff640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 0x7f8b3c0a5050 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c198fc0 tx=0x7f8b38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.397+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b4009c2f0 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/434407996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:20 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.582+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b4c02d170 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.582+0000 7f8b49ffb640 1 -- 172.21.15.107:0/554512098 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8b4c02d170 con 0x7f8b4c0ff7f0 2024-04-24T23:45:20.582 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 msgr2=0x7f8b3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 0x7f8b3c0a5050 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c198fc0 tx=0x7f8b38005f90 comp rx=0 tx=0).stop 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 msgr2=0x7f8b4c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000e9c0 tx=0x7f8b4000ee80 comp rx=0 tx=0).stop 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 shutdown_connections 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8b3c0a2b90 0x7f8b3c0a5050 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b4c1001b0 0x7f8b4c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 --2- 172.21.15.107:0/554512098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b4c0ff7f0 0x7f8b4c197ab0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 >> 172.21.15.107:0/554512098 conn(0x7f8b4c0fb370 msgr2=0x7f8b4c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 shutdown_connections 2024-04-24T23:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.584+0000 7f8b52fe9640 1 -- 172.21.15.107:0/554512098 wait complete. 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/434407996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:20 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:20.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1835319054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8103220 msgr2=0x7fa1d8109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1835319054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8103220 0x7fa1d8109ab0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009ca0 tx=0x7fa1d002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1835319054 shutdown_connections 2024-04-24T23:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1835319054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8103220 0x7fa1d8109ab0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1835319054 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 0x7fa1d8102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1835319054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8101670 0x7fa1d8101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1835319054 >> 172.21.15.107:0/1835319054 conn(0x7fa1d80fcbe0 msgr2=0x7fa1d80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1835319054 shutdown_connections 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.706+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1835319054 wait complete. 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 Processor -- start 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 -- start start 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 0x7fa1d8197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8103220 0x7fa1d819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d810d2d0 con 0x7fa1d8101670 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d810d150 con 0x7fa1d8103220 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dedfb640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d810d450 con 0x7fa1d8102860 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43050/0 (socket says 172.21.15.107:43050) 2024-04-24T23:45:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 -- 172.21.15.107:0/1801988605 learned_addr learned my addr 172.21.15.107:0/1801988605 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1de5fa640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8103220 0x7fa1d819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dd5f8640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 0x7fa1d8197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 -- 172.21.15.107:0/1801988605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 msgr2=0x7fa1d8197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 0x7fa1d8197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 -- 172.21.15.107:0/1801988605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8103220 msgr2=0x7fa1d819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8103220 0x7fa1d819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.707+0000 7fa1dddf9640 1 -- 172.21.15.107:0/1801988605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d819f500 con 0x7fa1d8101670 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1dddf9640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400e9c0 tx=0x7fa1d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d400cde0 con 0x7fa1d8101670 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d819f7f0 con 0x7fa1d8101670 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1d4010910 con 0x7fa1d8101670 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4018a10 con 0x7fa1d8101670 2024-04-24T23:45:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.708+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d819fd40 con 0x7fa1d8101670 2024-04-24T23:45:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.709+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d81089e0 con 0x7fa1d8101670 2024-04-24T23:45:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.710+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa1d401f020 con 0x7fa1d8101670 2024-04-24T23:45:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.710+0000 7fa1ceffd640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 0x7fa1c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:20.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.710+0000 7fa1dd5f8640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 0x7fa1c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:20.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.710+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa1d40cf620 con 0x7fa1d8101670 2024-04-24T23:45:20.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.711+0000 7fa1dd5f8640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 0x7fa1c40a5050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8198df0 tx=0x7fa1c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:20.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.714+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa1d409c1b0 con 0x7fa1d8101670 2024-04-24T23:45:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.897+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1d802d170 con 0x7fa1d8101670 2024-04-24T23:45:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.898+0000 7fa1ceffd640 1 -- 172.21.15.107:0/1801988605 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fa1d802d170 con 0x7fa1d8101670 2024-04-24T23:45:20.898 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 msgr2=0x7fa1c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 0x7fa1c40a5050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8198df0 tx=0x7fa1c800b410 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 msgr2=0x7fa1d81978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400e9c0 tx=0x7fa1d400ee80 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 shutdown_connections 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa1c40a2b90 0x7fa1c40a5050 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1d8103220 0x7fa1d819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa1d8102860 0x7fa1d8197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 --2- 172.21.15.107:0/1801988605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d8101670 0x7fa1d81978e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 >> 172.21.15.107:0/1801988605 conn(0x7fa1d80fcbe0 msgr2=0x7fa1d8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 shutdown_connections 2024-04-24T23:45:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:20.900+0000 7fa1dedfb640 1 -- 172.21.15.107:0/1801988605 wait complete. 2024-04-24T23:45:20.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:45:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 -- 172.21.15.107:0/4150178382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c102230 msgr2=0x7fbc2c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/4150178382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c102230 0x7fbc2c108ac0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009ca0 tx=0x7fbc2402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 -- 172.21.15.107:0/4150178382 shutdown_connections 2024-04-24T23:45:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/4150178382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c102230 0x7fbc2c108ac0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/4150178382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c101870 0x7fbc2c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/4150178382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c100680 0x7fbc2c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 -- 172.21.15.107:0/4150178382 >> 172.21.15.107:0/4150178382 conn(0x7fbc2c0fbbf0 msgr2=0x7fbc2c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 -- 172.21.15.107:0/4150178382 shutdown_connections 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.012+0000 7fbc32fc1640 1 -- 172.21.15.107:0/4150178382 wait complete. 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 Processor -- start 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 -- start start 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c100680 0x7fbc2c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c102230 0x7fbc2c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10d420 con 0x7fbc2c101870 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10d2a0 con 0x7fbc2c100680 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc32fc1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10d5a0 con 0x7fbc2c102230 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43070/0 (socket says 172.21.15.107:43070) 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 -- 172.21.15.107:0/496403144 learned_addr learned my addr 172.21.15.107:0/496403144 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc30d5d640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c100680 0x7fbc2c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc3155e640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c102230 0x7fbc2c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 -- 172.21.15.107:0/496403144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c102230 msgr2=0x7fbc2c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c102230 0x7fbc2c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 -- 172.21.15.107:0/496403144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c100680 msgr2=0x7fbc2c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c100680 0x7fbc2c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 -- 172.21.15.107:0/496403144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c19f690 con 0x7fbc2c101870 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc2bfff640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800d950 tx=0x7fbc1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.013+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18017070 con 0x7fbc2c101870 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.014+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc18009c30 con 0x7fbc2c101870 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.014+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c19f980 con 0x7fbc2c101870 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.014+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800f6b0 con 0x7fbc2c101870 2024-04-24T23:45:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.014+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c19ff20 con 0x7fbc2c101870 2024-04-24T23:45:21.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.015+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2c1079f0 con 0x7fbc2c101870 2024-04-24T23:45:21.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.016+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbc180109d0 con 0x7fbc2c101870 2024-04-24T23:45:21.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.016+0000 7fbc29ffb640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 0x7fbc1c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.016+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbc180cef50 con 0x7fbc2c101870 2024-04-24T23:45:21.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.016+0000 7fbc30d5d640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 0x7fbc1c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.017+0000 7fbc30d5d640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 0x7fbc1c0a50d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c1016d0 tx=0x7fbc20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.021+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc18009de0 con 0x7fbc2c101870 2024-04-24T23:45:21.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.198+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbc2c02d170 con 0x7fbc2c101870 2024-04-24T23:45:21.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.199+0000 7fbc29ffb640 1 -- 172.21.15.107:0/496403144 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbc1809b490 con 0x7fbc2c101870 2024-04-24T23:45:21.199 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 msgr2=0x7fbc1c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 0x7fbc1c0a50d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c1016d0 tx=0x7fbc20005f90 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 msgr2=0x7fbc2c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800d950 tx=0x7fbc1800de10 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 shutdown_connections 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbc1c0a2c10 0x7fbc1c0a50d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbc2c102230 0x7fbc2c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc2c101870 0x7fbc2c197fb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 --2- 172.21.15.107:0/496403144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2c100680 0x7fbc2c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 >> 172.21.15.107:0/496403144 conn(0x7fbc2c0fbbf0 msgr2=0x7fbc2c104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 shutdown_connections 2024-04-24T23:45:21.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.201+0000 7fbc32fc1640 1 -- 172.21.15.107:0/496403144 wait complete. 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: pgmap v369: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/929136741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/554512098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:21.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:21 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1801988605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- 172.21.15.107:0/2292910177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220100680 msgr2=0x7f4220100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 --2- 172.21.15.107:0/2292910177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220100680 0x7f4220100aa0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4210009900 tx=0x7f421002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- 172.21.15.107:0/2292910177 shutdown_connections 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 --2- 172.21.15.107:0/2292910177 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f4220108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 --2- 172.21.15.107:0/2292910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220101870 0x7f4220101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 --2- 172.21.15.107:0/2292910177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220100680 0x7f4220100aa0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- 172.21.15.107:0/2292910177 >> 172.21.15.107:0/2292910177 conn(0x7f42200fbbf0 msgr2=0x7f42200fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- 172.21.15.107:0/2292910177 shutdown_connections 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- 172.21.15.107:0/2292910177 wait complete. 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 Processor -- start 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.314+0000 7f422676a640 1 -- start start 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220100680 0x7f4220069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f422019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f422010c2d0 con 0x7f4220101870 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f422010c150 con 0x7f4220100680 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f422010c450 con 0x7f4220102230 2024-04-24T23:45:21.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43096/0 (socket says 172.21.15.107:43096) 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 -- 172.21.15.107:0/1647374256 learned_addr learned my addr 172.21.15.107:0/1647374256 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421ffff640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220100680 0x7f4220069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f4224d07640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f422019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 -- 172.21.15.107:0/1647374256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 msgr2=0x7f422019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f422019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 -- 172.21.15.107:0/1647374256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220100680 msgr2=0x7f4220069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220100680 0x7f4220069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 -- 172.21.15.107:0/1647374256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422019f780 con 0x7f4220101870 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f4224d07640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f422019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421f7fe640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f421400e9f0 tx=0x7f421400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421400cd80 con 0x7f4220101870 2024-04-24T23:45:21.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4214010910 con 0x7f4220101870 2024-04-24T23:45:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422019fa10 con 0x7f4220101870 2024-04-24T23:45:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4214018990 con 0x7f4220101870 2024-04-24T23:45:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.315+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422019ff60 con 0x7f4220101870 2024-04-24T23:45:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.316+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4220107a90 con 0x7f4220101870 2024-04-24T23:45:21.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.318+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4214010ac0 con 0x7f4220101870 2024-04-24T23:45:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.319+0000 7f421d7fa640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 0x7f42080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.319+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f42140cfc50 con 0x7f4220101870 2024-04-24T23:45:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.319+0000 7f421ffff640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 0x7f42080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.319+0000 7f421ffff640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 0x7f42080a5050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4210006010 tx=0x7f4210000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.322+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f421409c7e0 con 0x7f4220101870 2024-04-24T23:45:21.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.543+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f422002d170 con 0x7f4220101870 2024-04-24T23:45:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.544+0000 7f421d7fa640 1 -- 172.21.15.107:0/1647374256 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7f421409c0e0 con 0x7f4220101870 2024-04-24T23:45:21.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:21.544 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:45:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 msgr2=0x7f42080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 0x7f42080a5050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4210006010 tx=0x7f4210000bc0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 msgr2=0x7f422019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f421400e9f0 tx=0x7f421400eeb0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 shutdown_connections 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f42080a2b90 0x7f42080a5050 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4220102230 0x7f422019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4220101870 0x7f422019c850 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 --2- 172.21.15.107:0/1647374256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4220100680 0x7f4220069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 >> 172.21.15.107:0/1647374256 conn(0x7f42200fbbf0 msgr2=0x7f4220104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 shutdown_connections 2024-04-24T23:45:21.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.546+0000 7f422676a640 1 -- 172.21.15.107:0/1647374256 wait complete. 2024-04-24T23:45:21.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:45:21.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: pgmap v369: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/929136741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/554512098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:21 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1801988605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: pgmap v369: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/929136741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/554512098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:45:21.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:21 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1801988605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 -- 172.21.15.107:0/1936170083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc1017b0 msgr2=0x7f48cc101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 --2- 172.21.15.107:0/1936170083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc1017b0 0x7f48cc101bd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009900 tx=0x7f48c002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 -- 172.21.15.107:0/1936170083 shutdown_connections 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 --2- 172.21.15.107:0/1936170083 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc103360 0x7f48cc109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 --2- 172.21.15.107:0/1936170083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1029a0 0x7f48cc102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 --2- 172.21.15.107:0/1936170083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc1017b0 0x7f48cc101bd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 -- 172.21.15.107:0/1936170083 >> 172.21.15.107:0/1936170083 conn(0x7f48cc0fcd20 msgr2=0x7f48cc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 -- 172.21.15.107:0/1936170083 shutdown_connections 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.660+0000 7f48d4716640 1 -- 172.21.15.107:0/1936170083 wait complete. 2024-04-24T23:45:21.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 Processor -- start 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 -- start start 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 0x7f48cc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 0x7f48cc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10d400 con 0x7f48cc103360 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10d280 con 0x7f48cc1017b0 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d4716640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc10d580 con 0x7f48cc1029a0 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43126/0 (socket says 172.21.15.107:43126) 2024-04-24T23:45:21.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 -- 172.21.15.107:0/3937510061 learned_addr learned my addr 172.21.15.107:0/3937510061 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d1cb1640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 0x7f48cc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d24b2640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 0x7f48cc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 -- 172.21.15.107:0/3937510061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 msgr2=0x7f48cc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 0x7f48cc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 -- 172.21.15.107:0/3937510061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 msgr2=0x7f48cc197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 0x7f48cc197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 -- 172.21.15.107:0/3937510061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc19f700 con 0x7f48cc103360 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d1cb1640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 0x7f48cc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d24b2640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 0x7f48cc197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.661+0000 7f48d2cb3640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eaa0 tx=0x7f48c400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.662+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c400cdb0 con 0x7f48cc103360 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.662+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc19f9f0 con 0x7f48cc103360 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.662+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48c4005020 con 0x7f48cc103360 2024-04-24T23:45:21.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.662+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c40116b0 con 0x7f48cc103360 2024-04-24T23:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.662+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc19fef0 con 0x7f48cc103360 2024-04-24T23:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.663+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc108af0 con 0x7f48cc103360 2024-04-24T23:45:21.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.664+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f48c4005560 con 0x7f48cc103360 2024-04-24T23:45:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.664+0000 7f48bf7fe640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 0x7f48b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.665+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f48c40cfa80 con 0x7f48cc103360 2024-04-24T23:45:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.665+0000 7f48d24b2640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 0x7f48b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.665+0000 7f48d24b2640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 0x7f48b80a5050 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000c00 tx=0x7f48c0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.668+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f48c4016030 con 0x7f48cc103360 2024-04-24T23:45:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.848+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48cc062340 con 0x7f48cc103360 2024-04-24T23:45:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.850+0000 7f48bf7fe640 1 -- 172.21.15.107:0/3937510061 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7f48c409c610 con 0x7f48cc103360 2024-04-24T23:45:21.850 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:21.850 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 msgr2=0x7f48b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 0x7f48b80a5050 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000c00 tx=0x7f48c0031400 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 msgr2=0x7f48cc19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f48c400eaa0 tx=0x7f48c400ef60 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 shutdown_connections 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f48b80a2b90 0x7f48b80a5050 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48cc103360 0x7f48cc19f090 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f48cc1029a0 0x7f48cc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 --2- 172.21.15.107:0/3937510061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48cc1017b0 0x7f48cc197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 >> 172.21.15.107:0/3937510061 conn(0x7f48cc0fcd20 msgr2=0x7f48cc105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 shutdown_connections 2024-04-24T23:45:21.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.852+0000 7f48d4716640 1 -- 172.21.15.107:0/3937510061 wait complete. 2024-04-24T23:45:21.863 DEBUG:tasks.fwd_scrub:Yielding 2024-04-24T23:45:21.864 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T23:45:21.877 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:45:21.878 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T23:45:21.878 INFO:tasks.workunit:timeout=3h 2024-04-24T23:45:21.878 INFO:tasks.workunit:cleanup=True 2024-04-24T23:45:21.878 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:21.879 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-24T23:45:21.879 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T23:45:21.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:21.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-24 23:43:19.133718217 +0000 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-24 23:43:19.133718217 +0000 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-24 23:45:13.488442024 +0000 2024-04-24T23:45:21.895 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-24 23:43:19.133718217 +0000 2024-04-24T23:45:21.895 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-24T23:45:21.896 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-24T23:45:21.964 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:45:21.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 -- 172.21.15.107:0/3189756917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84100680 msgr2=0x7f3b84100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3189756917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84100680 0x7f3b84100aa0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009900 tx=0x7f3b7802a410 comp rx=0 tx=0).stop 2024-04-24T23:45:21.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 -- 172.21.15.107:0/3189756917 shutdown_connections 2024-04-24T23:45:21.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3189756917 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 0x7f3b84108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3189756917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84101870 0x7f3b84101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.984+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3189756917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84100680 0x7f3b84100aa0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- 172.21.15.107:0/3189756917 >> 172.21.15.107:0/3189756917 conn(0x7f3b840fbbf0 msgr2=0x7f3b840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- 172.21.15.107:0/3189756917 shutdown_connections 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- 172.21.15.107:0/3189756917 wait complete. 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 Processor -- start 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- start start 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 0x7f3b841969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 0x7f3b8419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8410c2d0 con 0x7f3b84101870 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8410c150 con 0x7f3b84100680 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.985+0000 7f3b89df9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8410c450 con 0x7f3b84102230 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43144/0 (socket says 172.21.15.107:43144) 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 -- 172.21.15.107:0/3708285272 learned_addr learned my addr 172.21.15.107:0/3708285272 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b83fff640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 0x7f3b8419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 0x7f3b841969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 -- 172.21.15.107:0/3708285272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 msgr2=0x7f3b8419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 0x7f3b8419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 -- 172.21.15.107:0/3708285272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 msgr2=0x7f3b841969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:21.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 0x7f3b841969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 -- 172.21.15.107:0/3708285272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8419e5c0 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 0x7f3b841969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c920 tx=0x7f3b7400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b80ff9640 1 -- 172.21.15.107:0/3708285272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74007c70 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8419e8b0 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b80ff9640 1 -- 172.21.15.107:0/3708285272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b74007e10 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8419ee50 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.986+0000 7f3b80ff9640 1 -- 172.21.15.107:0/3708285272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74018890 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.989+0000 7f3b80ff9640 1 -- 172.21.15.107:0/3708285272 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3b74018b10 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.989+0000 7f3b80ff9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 0x7f3b6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b80ff9640 1 -- 172.21.15.107:0/3708285272 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3b740cf560 con 0x7f3b84101870 2024-04-24T23:45:21.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 0x7f3b6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 Processor -- start 2024-04-24T23:45:21.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 -- start start 2024-04-24T23:45:21.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a88e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a895f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f00a7ef2c0 con 0x55f00a88e990 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55f00a797910 con 0x55f00a893b00 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55f00a7eb280 con 0x55f00a88dfa0 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43158/0 (socket says 172.21.15.107:43158) 2024-04-24T23:45:21.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 -- 172.21.15.107:0/1656297948 learned_addr learned my addr 172.21.15.107:0/1656297948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 0x7f3b6c0a5050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3b78006010 tx=0x7f3b78032dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b83fff640 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a895f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 -- 172.21.15.107:0/1656297948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 msgr2=0x55f00a88e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a88e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 -- 172.21.15.107:0/1656297948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 msgr2=0x55f00a895f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a895f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 -- 172.21.15.107:0/1656297948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f00a7a8fb0 con 0x55f00a88e990 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b83fff640 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a895f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.990+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3b740a4130 tx=0x7f3b7409c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403a0b7044a5fab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b69ffb640 1 -- 172.21.15.107:0/1656297948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74099a90 con 0x55f00a88e990 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b69ffb640 1 -- 172.21.15.107:0/1656297948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b74099cf0 con 0x55f00a88e990 2024-04-24T23:45:22.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b69ffb640 1 -- 172.21.15.107:0/1656297948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740a41d0 con 0x55f00a88e990 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/1656297948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 msgr2=0x55f00a893490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3b740a4130 tx=0x7f3b7409c230 comp rx=0 tx=0).stop 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/1656297948 shutdown_connections 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a895f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a893490 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/1656297948 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a88e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/1656297948 >> 172.21.15.107:0/1656297948 conn(0x55f00a853950 msgr2=0x55f00a887810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/1656297948 shutdown_connections 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.991+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/1656297948 wait complete. 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 -- start start 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 Processor -- start 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a898480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a89fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f00a896ab0 con 0x55f00a88e990 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55f00a7aae90 con 0x55f00a893b00 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55f00a896c30 con 0x55f00a88dfa0 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b8cbe2740 10 client.0 _ll_register_callbacks cb 0x55f00a7e6970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43168/0 (socket says 172.21.15.107:43168) 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b82ffd640 1 -- 172.21.15.107:0/725535611 learned_addr learned my addr 172.21.15.107:0/725535611 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b837fe640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a898480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.992+0000 7f3b83fff640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a89fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 -- 172.21.15.107:0/725535611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 msgr2=0x55f00a898480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a898480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 -- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 msgr2=0x55f00a89fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a89fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 -- 172.21.15.107:0/725535611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00a5cb7e0 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b837fe640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a898480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3b7409b980 tx=0x7f3b740a4720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dfd4876a7725d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740b8030 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3b740ba2d0 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b74099cf0 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3b740a41d0 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 10 client.15678 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f00a62bf70 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b74099cf0 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3b74097740 con 0x55f00a88e990 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 10 client.15678 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:22.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 10 client.15678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.993+0000 7f3b8cbe2740 20 client.15678 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a8a3b60 0x55f00a8a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b8cbe2740 4 client.15678 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f00a7c35a0 con 0x55f00a8a3b60 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b837fe640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a8a3b60 0x55f00a8a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b837fe640 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a8a3b60 0x55f00a8a5f50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f4875e71f7f514 server_cookie=202aa8aa20a4636f in_seq=0 out_seq=0 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.994+0000 7f3b62ffd640 10 client.15678 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b62ffd640 1 -- 172.21.15.107:0/725535611 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f00a7c35a0 con 0x55f00a8a3b60 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b62ffd640 10 client.15678 handle_command_reply: tid=0 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 client.15678 shutdown 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 msgr2=0x55f00a8989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3b7409b980 tx=0x7f3b740a4720 comp rx=0 tx=0).stop 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 shutdown_connections 2024-04-24T23:45:22.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a8a3b60 0x55f00a8a5f50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a893b00 0x55f00a89fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a8989c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b62ffd640 0 client.15678 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/725535611 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88dfa0 0x55f00a898480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 >> 172.21.15.107:0/725535611 conn(0x55f00a853950 msgr2=0x55f00a886950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 shutdown_connections 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/725535611 wait complete. 2024-04-24T23:45:22.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:21.995+0000 7f3b8cbe2740 20 client.15678 trim_cache size 0 max 16384 2024-04-24T23:45:22.005 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T23:45:22.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 Processor -- start 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 -- start start 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a88af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a893b00 0x55f00a8860c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a8340 con 0x55f00a88e990 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a9e80 con 0x55f00a88dfa0 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a8a10 con 0x55f00a893b00 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43172/0 (socket says 172.21.15.107:43172) 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b82ffd640 1 -- 172.21.15.107:0/109705878 learned_addr learned my addr 172.21.15.107:0/109705878 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b837fe640 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a88af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.013+0000 7f3b83fff640 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a893b00 0x55f00a8860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 -- 172.21.15.107:0/109705878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a893b00 msgr2=0x55f00a8860c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a893b00 0x55f00a8860c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 -- 172.21.15.107:0/109705878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 msgr2=0x55f00a88af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a88af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 -- 172.21.15.107:0/109705878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f00a4c3170 con 0x55f00a88e990 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3b74097e50 tx=0x7f3b7400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513e6376e8efff55 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b697fa640 1 -- 172.21.15.107:0/109705878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740a4370 con 0x55f00a88e990 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b697fa640 1 -- 172.21.15.107:0/109705878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b740ba2d0 con 0x55f00a88e990 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.014+0000 7f3b697fa640 1 -- 172.21.15.107:0/109705878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7409ae40 con 0x55f00a88e990 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/109705878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 msgr2=0x55f00a89f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3b74097e50 tx=0x7f3b7400fd40 comp rx=0 tx=0).stop 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/109705878 shutdown_connections 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a893b00 0x55f00a8860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a88e990 0x55f00a89f9c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/109705878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a88af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/109705878 >> 172.21.15.107:0/109705878 conn(0x55f00a853950 msgr2=0x55f00a8aa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/109705878 shutdown_connections 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.015+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/109705878 wait complete. 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 -- start start 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 Processor -- start 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a89f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 0x55f00a89f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a97b0 con 0x55f00a893b00 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a90e0 con 0x55f00a88dfa0 2024-04-24T23:45:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841a6320 con 0x55f00a88e990 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43188/0 (socket says 172.21.15.107:43188) 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b83fff640 1 -- 172.21.15.107:0/3826103682 learned_addr learned my addr 172.21.15.107:0/3826103682 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.016+0000 7f3b8cbe2740 10 client.0 _ll_register_callbacks cb 0x7f3b7409b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 0x55f00a89f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a89f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 -- 172.21.15.107:0/3826103682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 msgr2=0x55f00a89f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 0x55f00a89f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 -- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 msgr2=0x55f00a89f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a89f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 -- 172.21.15.107:0/3826103682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00a89e080 con 0x55f00a893b00 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b82ffd640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 0x55f00a89f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b83fff640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c003c20 tx=0x7f3b7c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990cb6e43f463416 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c020060 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3b7c011850 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b740ba2d0 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3b7c027050 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b8cbe2740 10 client.15684 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f00a90a9a0 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.017+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b740ba2d0 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3b7c01a070 con 0x55f00a893b00 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 10 client.15684 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 10 client.15684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 20 client.15684 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a90b3b0 0x55f00a90d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 4 client.15684 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T23:45:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f00a7fc800 con 0x55f00a90b3b0 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a90b3b0 0x55f00a90d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b837fe640 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a90b3b0 0x55f00a90d7d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82231801937005f7 server_cookie=b80703dac902f843 in_seq=0 out_seq=0 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.018+0000 7f3b62ffd640 10 client.15684 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b62ffd640 1 -- 172.21.15.107:0/3826103682 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f00a7fc800 con 0x55f00a90b3b0 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b62ffd640 10 client.15684 handle_command_reply: tid=0 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 client.15684 shutdown 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 msgr2=0x55f00a89d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c003c20 tx=0x7f3b7c004410 comp rx=0 tx=0).stop 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 shutdown_connections 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55f00a90b3b0 0x55f00a90d7d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b62ffd640 0 client.15684 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f00a893b00 0x55f00a89d940 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55f00a88e990 0x55f00a89f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 --2- 172.21.15.107:0/3826103682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55f00a88dfa0 0x55f00a89f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 >> 172.21.15.107:0/3826103682 conn(0x55f00a853950 msgr2=0x55f00a88ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 shutdown_connections 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.020+0000 7f3b8cbe2740 1 -- 172.21.15.107:0/3826103682 wait complete. 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b8cbe2740 20 client.15684 trim_cache size 0 max 16384 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "f9424706-27d2-40cc-b260-ca66812fc4a9", 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-04-24T23:45:22.027 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 msgr2=0x7f3b6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 0x7f3b6c0a5050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3b78006010 tx=0x7f3b78032dd0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 msgr2=0x7f3b84196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c920 tx=0x7f3b7400cde0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 shutdown_connections 2024-04-24T23:45:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3b6c0a2b90 0x7f3b6c0a5050 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3b84102230 0x7f3b8419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b84101870 0x7f3b84196ee0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 --2- 172.21.15.107:0/3708285272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b84100680 0x7f3b841969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 >> 172.21.15.107:0/3708285272 conn(0x7f3b840fbbf0 msgr2=0x7f3b84104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 shutdown_connections 2024-04-24T23:45:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.021+0000 7f3b89df9640 1 -- 172.21.15.107:0/3708285272 wait complete. 2024-04-24T23:45:22.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:45:22.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:45:22.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 -- 172.21.15.107:0/3865150683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4102230 msgr2=0x7fedd4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 --2- 172.21.15.107:0/3865150683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4102230 0x7fedd4108ac0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009ca0 tx=0x7fedcc02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 -- 172.21.15.107:0/3865150683 shutdown_connections 2024-04-24T23:45:22.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 --2- 172.21.15.107:0/3865150683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4102230 0x7fedd4108ac0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 --2- 172.21.15.107:0/3865150683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4101870 0x7fedd4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 --2- 172.21.15.107:0/3865150683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 -- 172.21.15.107:0/3865150683 >> 172.21.15.107:0/3865150683 conn(0x7fedd40fbbf0 msgr2=0x7fedd40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 -- 172.21.15.107:0/3865150683 shutdown_connections 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.177+0000 7feddaaba640 1 -- 172.21.15.107:0/3865150683 wait complete. 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 Processor -- start 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 -- start start 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4102230 0x7fedd419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fedd410c300 con 0x7fedd4101870 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fedd410c180 con 0x7fedd4100680 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7feddaaba640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fedd410c480 con 0x7fedd4102230 2024-04-24T23:45:22.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43194/0 (socket says 172.21.15.107:43194) 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 -- 172.21.15.107:0/1919317322 learned_addr learned my addr 172.21.15.107:0/1919317322 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd9057640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4102230 0x7fedd419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd8856640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 -- 172.21.15.107:0/1919317322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4102230 msgr2=0x7fedd419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4102230 0x7fedd419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 -- 172.21.15.107:0/1919317322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 msgr2=0x7fedd4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 -- 172.21.15.107:0/1919317322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd419f6b0 con 0x7fedd4101870 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd8856640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd3fff640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fedc000e9f0 tx=0x7fedc000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd1ffb640 1 -- 172.21.15.107:0/1919317322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc000cd80 con 0x7fedd4101870 2024-04-24T23:45:22.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.178+0000 7fedd1ffb640 1 -- 172.21.15.107:0/1919317322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc0005500 con 0x7fedd4101870 2024-04-24T23:45:22.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.179+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd419f9a0 con 0x7fedd4101870 2024-04-24T23:45:22.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.179+0000 7fedd1ffb640 1 -- 172.21.15.107:0/1919317322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0010690 con 0x7fedd4101870 2024-04-24T23:45:22.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.179+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd419fea0 con 0x7fedd4101870 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.181+0000 7fedd1ffb640 1 -- 172.21.15.107:0/1919317322 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fedc00046c0 con 0x7fedd4101870 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.181+0000 7fedd1ffb640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 0x7fedc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.181+0000 7fedd1ffb640 1 -- 172.21.15.107:0/1919317322 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fedc00cf970 con 0x7fedd4101870 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.181+0000 7fedd8856640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 0x7fedc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7fedd8856640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 0x7fedc40a5050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fedd41016d0 tx=0x7fedc8005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 Processor -- start 2024-04-24T23:45:22.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 -- start start 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2d8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55800a2376b0 con 0x55800a2d8da0 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55800a1df880 con 0x55800a2d3240 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7feddd7ca740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55800a233670 con 0x55800a2d3c30 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.182+0000 7fedd9057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43206/0 (socket says 172.21.15.107:43206) 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 -- 172.21.15.107:0/206106207 learned_addr learned my addr 172.21.15.107:0/206106207 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd8856640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd3fff640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2d8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 -- 172.21.15.107:0/206106207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 msgr2=0x55800a2d8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2d8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 -- 172.21.15.107:0/206106207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 msgr2=0x55800a2d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 -- 172.21.15.107:0/206106207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800a0157d0 con 0x55800a2d8da0 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd3fff640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2d8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedd9057640 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fedcc035000 tx=0x7fedcc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57561d9e5034c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedbeffd640 1 -- 172.21.15.107:0/206106207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc031cb0 con 0x55800a2d8da0 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedbeffd640 1 -- 172.21.15.107:0/206106207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedcc036050 con 0x55800a2d8da0 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.183+0000 7fedbeffd640 1 -- 172.21.15.107:0/206106207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0043d0 con 0x55800a2d8da0 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- 172.21.15.107:0/206106207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 msgr2=0x55800a2db230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fedcc035000 tx=0x7fedcc009360 comp rx=0 tx=0).stop 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- 172.21.15.107:0/206106207 shutdown_connections 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2db230 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2d8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 --2- 172.21.15.107:0/206106207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- 172.21.15.107:0/206106207 >> 172.21.15.107:0/206106207 conn(0x55800a2ce8b0 msgr2=0x55800a2cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- 172.21.15.107:0/206106207 shutdown_connections 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- 172.21.15.107:0/206106207 wait complete. 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 -- start start 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.184+0000 7feddd7ca740 1 Processor -- start 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2dd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d8da0 0x55800a2e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55800a2dbb70 con 0x55800a2d3c30 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55800a1f2f50 con 0x55800a2d3240 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55800a2dbcf0 con 0x55800a2d8da0 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 10 client.0 _ll_register_callbacks cb 0x55800a22e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd8856640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2dd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43216/0 (socket says 172.21.15.107:43216) 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 -- 172.21.15.107:0/313926281 learned_addr learned my addr 172.21.15.107:0/313926281 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd9057640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d8da0 0x55800a2e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 -- 172.21.15.107:0/313926281 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d8da0 msgr2=0x55800a2e4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d8da0 0x55800a2e4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 -- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 msgr2=0x55800a2dd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2dd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 -- 172.21.15.107:0/313926281 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558009f4b3f0 con 0x55800a2d3c30 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedd3fff640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fedc00987d0 tx=0x7fedc009c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f11cef8f84a8b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc00a4a70 con 0x55800a2d3c30 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fedc0016df0 con 0x55800a2d3c30 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.185+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedcc036050 con 0x55800a2d3c30 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fedc00a45e0 con 0x55800a2d3c30 2024-04-24T23:45:22.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558009f0b0f0 con 0x55800a2d3c30 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedcc036050 con 0x55800a2d3c30 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fedc009acc0 con 0x55800a2d3c30 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 10 client.15696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 20 client.15696 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a2e92d0 0x55800a2eb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 4 client.15696 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55800a244bf0 con 0x55800a2e92d0 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.186+0000 7fedd8856640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a2e92d0 0x55800a2eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7fedd8856640 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a2e92d0 0x55800a2eb790 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24beb81e37593ba9 server_cookie=a402717328c559aa in_seq=0 out_seq=0 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7fedb3fff640 10 client.15696 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7fedb3fff640 1 -- 172.21.15.107:0/313926281 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55800a244bf0 con 0x55800a2e92d0 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7fedb3fff640 10 client.15696 handle_command_reply: tid=0 2024-04-24T23:45:22.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 client.15696 shutdown 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 msgr2=0x55800a2dd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fedc00987d0 tx=0x7fedc009c500 comp rx=0 tx=0).stop 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 shutdown_connections 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a2e92d0 0x55800a2eb790 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d8da0 0x55800a2e4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3c30 0x55800a2dd9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 --2- 172.21.15.107:0/313926281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2dd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7fedb3fff640 0 client.15696 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.187+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 >> 172.21.15.107:0/313926281 conn(0x55800a2ce8b0 msgr2=0x55800a2cdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.188+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 shutdown_connections 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.188+0000 7feddd7ca740 1 -- 172.21.15.107:0/313926281 wait complete. 2024-04-24T23:45:22.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.188+0000 7feddd7ca740 20 client.15696 trim_cache size 0 max 16384 2024-04-24T23:45:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 Processor -- start 2024-04-24T23:45:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 -- start start 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d8da0 0x55800a2d5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41aa460 con 0x55800a2d3240 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41abfa0 con 0x55800a2d8da0 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7feddd7ca740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41aab30 con 0x55800a2d3c30 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43230/0 (socket says 172.21.15.107:43230) 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 -- 172.21.15.107:0/3030649790 learned_addr learned my addr 172.21.15.107:0/3030649790 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd3fff640 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 -- 172.21.15.107:0/3030649790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 msgr2=0x55800a2e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 -- 172.21.15.107:0/3030649790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d8da0 msgr2=0x55800a2d5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd9057640 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d8da0 0x55800a2d5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d8da0 0x55800a2d5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.205+0000 7fedd8856640 1 -- 172.21.15.107:0/3030649790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800a2d4010 con 0x55800a2d3240 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.206+0000 7fedd8856640 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005b60 tx=0x7fedc8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8fee3cd4847188 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.206+0000 7fedbe7fc640 1 -- 172.21.15.107:0/3030649790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8028530 con 0x55800a2d3240 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.206+0000 7fedbe7fc640 1 -- 172.21.15.107:0/3030649790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8017070 con 0x55800a2d3240 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.206+0000 7fedbe7fc640 1 -- 172.21.15.107:0/3030649790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc80103d0 con 0x55800a2d3240 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- 172.21.15.107:0/3030649790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 msgr2=0x55800a2d4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005b60 tx=0x7fedc8014000 comp rx=0 tx=0).stop 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- 172.21.15.107:0/3030649790 shutdown_connections 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d8da0 0x55800a2d5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- 172.21.15.107:0/3030649790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d3240 0x55800a2d4ff0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- 172.21.15.107:0/3030649790 >> 172.21.15.107:0/3030649790 conn(0x55800a2ce8b0 msgr2=0x55800a2dd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- 172.21.15.107:0/3030649790 shutdown_connections 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- 172.21.15.107:0/3030649790 wait complete. 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- start start 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 Processor -- start 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41ab8d0 con 0x55800a2d8da0 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41ab200 con 0x55800a2d3240 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fedd41a8440 con 0x55800a2d3c30 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.207+0000 7feddd7ca740 10 client.0 _ll_register_callbacks cb 0x7fedc00056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43236/0 (socket says 172.21.15.107:43236) 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 -- 172.21.15.107:0/1121092004 learned_addr learned my addr 172.21.15.107:0/1121092004 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd3fff640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd8856640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 -- 172.21.15.107:0/1121092004 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 msgr2=0x55800a2e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 -- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 msgr2=0x55800a2e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2e1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 -- 172.21.15.107:0/1121092004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800a2e3930 con 0x55800a2d8da0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedd9057640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x55800a2cf820 tx=0x7fedcc031990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d955ff22d46e188a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc02af40 con 0x55800a2d8da0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fedcc003b80 con 0x55800a2d8da0 2024-04-24T23:45:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc8017070 con 0x55800a2d8da0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fedcc003d80 con 0x55800a2d8da0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7feddd7ca740 10 client.15702 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.208+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55800a2e4c40 con 0x55800a2d8da0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8017070 con 0x55800a2d8da0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fedcc042070 con 0x55800a2d8da0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 10 client.15702 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 20 client.15702 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a34fd60 0x55800a352180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 4 client.15702 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55800a23ef40 con 0x55800a34fd60 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7fedd8856640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a34fd60 0x55800a352180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7fedd8856640 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a34fd60 0x55800a352180 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa1ad46f66793268 server_cookie=624c4fed76f7503a in_seq=0 out_seq=0 2024-04-24T23:45:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.209+0000 7fedb3fff640 10 client.15702 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7fedb3fff640 1 -- 172.21.15.107:0/1121092004 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55800a23ef40 con 0x55800a34fd60 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7fedb3fff640 10 client.15702 handle_command_reply: tid=0 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 client.15702 shutdown 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 msgr2=0x55800a2e3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x55800a2cf820 tx=0x7fedcc031990 comp rx=0 tx=0).stop 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 shutdown_connections 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55800a34fd60 0x55800a352180 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55800a2d8da0 0x55800a2e3340 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7fedb3fff640 0 client.15702 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55800a2d3c30 0x55800a2e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 --2- 172.21.15.107:0/1121092004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55800a2d3240 0x55800a2e1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 >> 172.21.15.107:0/1121092004 conn(0x55800a2ce8b0 msgr2=0x55800a2dd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 shutdown_connections 2024-04-24T23:45:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 1 -- 172.21.15.107:0/1121092004 wait complete. 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddd7ca740 20 client.15702 trim_cache size 0 max 16384 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "f9424706-27d2-40cc-b260-ca66812fc4a9": { 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "f9424706-27d2-40cc-b260-ca66812fc4a9", 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-04-24T23:45:22.217 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:45:22.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:45:22.218 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 msgr2=0x7fedc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 0x7fedc40a5050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fedd41016d0 tx=0x7fedc8005e70 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 msgr2=0x7fedd4198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.210+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fedc000e9f0 tx=0x7fedc000edf0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 shutdown_connections 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fedc40a2b90 0x7fedc40a5050 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fedd4102230 0x7fedd419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd4101870 0x7fedd4198020 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 --2- 172.21.15.107:0/1919317322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedd4100680 0x7fedd4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 >> 172.21.15.107:0/1919317322 conn(0x7fedd40fbbf0 msgr2=0x7fedd40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 shutdown_connections 2024-04-24T23:45:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:22.211+0000 7feddaaba640 1 -- 172.21.15.107:0/1919317322 wait complete. 2024-04-24T23:45:22.221 INFO:tasks.cephfs.filesystem:scrub status for tag:f9424706-27d2-40cc-b260-ca66812fc4a9 - {'path': '/', 'tag': 'f9424706-27d2-40cc-b260-ca66812fc4a9', 'options': 'recursive,force'} 2024-04-24T23:45:22.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:22 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/496403144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:22.460 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:22 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1647374256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:22.460 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:22 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3937510061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:22.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:22 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/496403144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:22.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:22 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1647374256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:22.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:22 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3937510061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:22.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:22 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/496403144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:45:22.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:22 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1647374256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:22.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:22 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3937510061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:45:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:23 smithi195 ceph-mon[33019]: pgmap v370: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-24T23:45:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:23 smithi195 ceph-mon[33019]: scrub queued for path: / 2024-04-24T23:45:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:23 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:23 smithi195 ceph-mon[33019]: scrub summary: active paths [/] 2024-04-24T23:45:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:23 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:23 smithi170 ceph-mon[32714]: pgmap v370: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-24T23:45:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:23 smithi170 ceph-mon[32714]: scrub queued for path: / 2024-04-24T23:45:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:23 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:23 smithi170 ceph-mon[32714]: scrub summary: active paths [/] 2024-04-24T23:45:23.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:23 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:23.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:23 smithi107 ceph-mon[29118]: pgmap v370: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-24T23:45:23.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:23 smithi107 ceph-mon[29118]: scrub queued for path: / 2024-04-24T23:45:23.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:23 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:23.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:23 smithi107 ceph-mon[29118]: scrub summary: active paths [/] 2024-04-24T23:45:23.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:23 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:25 smithi195 ceph-mon[33019]: pgmap v371: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:25 smithi170 ceph-mon[32714]: pgmap v371: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:25.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:25 smithi107 ceph-mon[29118]: pgmap v371: 129 pgs: 129 active+clean; 651 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T23:45:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:27 smithi195 ceph-mon[33019]: pgmap v372: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-24T23:45:27.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:27 smithi170 ceph-mon[32714]: pgmap v372: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-24T23:45:27.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:27 smithi107 ceph-mon[29118]: pgmap v372: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-24T23:45:29.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:29 smithi195 ceph-mon[33019]: pgmap v373: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:45:29.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:29 smithi170 ceph-mon[32714]: pgmap v373: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:45:29.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:29 smithi107 ceph-mon[29118]: pgmap v373: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:45:31.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:31 smithi195 ceph-mon[33019]: pgmap v374: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:31.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:31 smithi170 ceph-mon[32714]: pgmap v374: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:31.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:31 smithi107 ceph-mon[29118]: pgmap v374: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:33 smithi195 ceph-mon[33019]: pgmap v375: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:33 smithi170 ceph-mon[32714]: pgmap v375: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:33.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:33 smithi107 ceph-mon[29118]: pgmap v375: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-24T23:45:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:35 smithi195 ceph-mon[33019]: pgmap v376: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-24T23:45:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:35 smithi195 ceph-mon[33019]: scrub summary: idle 2024-04-24T23:45:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:35 smithi170 ceph-mon[32714]: pgmap v376: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-24T23:45:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:35 smithi170 ceph-mon[32714]: scrub summary: idle 2024-04-24T23:45:35.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:35 smithi107 ceph-mon[29118]: pgmap v376: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-24T23:45:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:35 smithi107 ceph-mon[29118]: scrub summary: idle 2024-04-24T23:45:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:37 smithi195 ceph-mon[33019]: pgmap v377: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:45:37.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:37 smithi170 ceph-mon[32714]: pgmap v377: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:45:37.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:37 smithi107 ceph-mon[29118]: pgmap v377: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:45:39.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:39 smithi195 ceph-mon[33019]: pgmap v378: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:45:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:39 smithi170 ceph-mon[32714]: pgmap v378: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:45:39.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:39 smithi107 ceph-mon[29118]: pgmap v378: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:45:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:41 smithi195 ceph-mon[33019]: pgmap v379: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:41 smithi170 ceph-mon[32714]: pgmap v379: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:41.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:41 smithi107 ceph-mon[29118]: pgmap v379: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:43 smithi195 ceph-mon[33019]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:43.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:43 smithi170 ceph-mon[32714]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:43.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:43 smithi107 ceph-mon[29118]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:45 smithi195 ceph-mon[33019]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:45 smithi170 ceph-mon[32714]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:45.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:45 smithi107 ceph-mon[29118]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:47 smithi195 ceph-mon[33019]: pgmap v382: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:47 smithi170 ceph-mon[32714]: pgmap v382: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:47.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:47 smithi107 ceph-mon[29118]: pgmap v382: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:45:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:49 smithi195 ceph-mon[33019]: pgmap v383: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:49 smithi170 ceph-mon[32714]: pgmap v383: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:49.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:49 smithi107 ceph-mon[29118]: pgmap v383: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:51 smithi195 ceph-mon[33019]: pgmap v384: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:51 smithi170 ceph-mon[32714]: pgmap v384: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:51.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:51 smithi107 ceph-mon[29118]: pgmap v384: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:45:52.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:45:52.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:45:52.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 -- 172.21.15.107:0/306023272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc104b30 msgr2=0x7f76bc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 --2- 172.21.15.107:0/306023272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc104b30 0x7f76bc106f20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009900 tx=0x7f76b002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:52.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 -- 172.21.15.107:0/306023272 shutdown_connections 2024-04-24T23:45:52.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 --2- 172.21.15.107:0/306023272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc107460 0x7f76bc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 --2- 172.21.15.107:0/306023272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc104b30 0x7f76bc106f20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 --2- 172.21.15.107:0/306023272 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc102200 0x7f76bc1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 -- 172.21.15.107:0/306023272 >> 172.21.15.107:0/306023272 conn(0x7f76bc0fbbf0 msgr2=0x7f76bc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 -- 172.21.15.107:0/306023272 shutdown_connections 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 -- 172.21.15.107:0/306023272 wait complete. 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.326+0000 7f76c202c640 1 Processor -- start 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 -- start start 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc104b30 0x7f76bc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc107460 0x7f76bc19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10d4b0 con 0x7f76bc102200 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10d330 con 0x7f76bc104b30 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76c202c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc10d630 con 0x7f76bc107460 2024-04-24T23:45:52.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51240/0 (socket says 172.21.15.107:51240) 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 -- 172.21.15.107:0/3135892598 learned_addr learned my addr 172.21.15.107:0/3135892598 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76baffd640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc104b30 0x7f76bc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bbfff640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc107460 0x7f76bc19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 -- 172.21.15.107:0/3135892598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc107460 msgr2=0x7f76bc19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc107460 0x7f76bc19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 -- 172.21.15.107:0/3135892598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc104b30 msgr2=0x7f76bc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc104b30 0x7f76bc198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 -- 172.21.15.107:0/3135892598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc19f740 con 0x7f76bc102200 2024-04-24T23:45:52.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.327+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00b4a0 tx=0x7f76ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.328+0000 7f76b8ff9640 1 -- 172.21.15.107:0/3135892598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac014070 con 0x7f76bc102200 2024-04-24T23:45:52.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.328+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc19fa30 con 0x7f76bc102200 2024-04-24T23:45:52.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.328+0000 7f76b8ff9640 1 -- 172.21.15.107:0/3135892598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ac010af0 con 0x7f76bc102200 2024-04-24T23:45:52.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.328+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc19ffd0 con 0x7f76bc102200 2024-04-24T23:45:52.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.328+0000 7f76b8ff9640 1 -- 172.21.15.107:0/3135892598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac017d10 con 0x7f76bc102200 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.330+0000 7f76c4e0e740 1 Processor -- start 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 -- start start 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76b8ff9640 1 -- 172.21.15.107:0/3135892598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f76ac017070 con 0x7f76bc102200 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa130660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa135730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5624fa0946b0 con 0x5624fa135da0 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5624fa03c880 con 0x5624fa130c30 2024-04-24T23:45:52.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5624fa090670 con 0x5624fa130240 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51252/0 (socket says 172.21.15.107:51252) 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 -- 172.21.15.107:0/215549761 learned_addr learned my addr 172.21.15.107:0/215549761 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76bbfff640 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76b8ff9640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 0x7f76a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 -- 172.21.15.107:0/215549761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 msgr2=0x5624fa130660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa130660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 -- 172.21.15.107:0/215549761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 msgr2=0x5624fa135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa135730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76baffd640 1 -- 172.21.15.107:0/215549761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624f9e727d0 con 0x5624fa135da0 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76bbfff640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 0x7f76a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.331+0000 7f76b8ff9640 1 -- 172.21.15.107:0/3135892598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f76ac0ce890 con 0x7f76bc102200 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76baffd640 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f76b0000c00 tx=0x7f76b0004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2924d8a7bfc147d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76a1ffb640 1 -- 172.21.15.107:0/215549761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b003f070 con 0x5624fa135da0 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76a1ffb640 1 -- 172.21.15.107:0/215549761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b0005520 con 0x5624fa135da0 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76a1ffb640 1 -- 172.21.15.107:0/215549761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0035030 con 0x5624fa135da0 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76c4e0e740 1 -- 172.21.15.107:0/215549761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 msgr2=0x5624fa138230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f76b0000c00 tx=0x7f76b0004190 comp rx=0 tx=0).stop 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.332+0000 7f76bbfff640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 0x7f76a40a5050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x5624fa131490 tx=0x7f76b4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- 172.21.15.107:0/215549761 shutdown_connections 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa135da0 0x5624fa138230 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa135730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/215549761 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa130660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- 172.21.15.107:0/215549761 >> 172.21.15.107:0/215549761 conn(0x5624fa12b8b0 msgr2=0x5624fa12bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- 172.21.15.107:0/215549761 shutdown_connections 2024-04-24T23:45:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- 172.21.15.107:0/215549761 wait complete. 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- start start 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 Processor -- start 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa13a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5624fa138cf0 con 0x5624fa130c30 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5624fa04ff50 con 0x5624fa135da0 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5624fa138e70 con 0x5624fa130240 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76c4e0e740 10 client.0 _ll_register_callbacks cb 0x5624fa08b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51256/0 (socket says 172.21.15.107:51256) 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.333+0000 7f76baffd640 1 -- 172.21.15.107:0/2358488442 learned_addr learned my addr 172.21.15.107:0/2358488442 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76bbfff640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 -- 172.21.15.107:0/2358488442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 msgr2=0x5624fa13a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa13a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 -- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 msgr2=0x5624fa141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 -- 172.21.15.107:0/2358488442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624f9da83f0 con 0x5624fa130c30 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa13a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76baffd640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76b002aa00 tx=0x7f76b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385f5dab1880518d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f769affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b004c020 con 0x5624fa130c30 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f76b0005020 con 0x5624fa130c30 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76b0005520 con 0x5624fa130c30 2024-04-24T23:45:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76b0057050 con 0x5624fa130c30 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76c4e0e740 10 client.15714 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.334+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5624f9d680f0 con 0x5624fa130c30 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b0005520 con 0x5624fa130c30 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f76b0003780 con 0x5624fa130c30 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 10 client.15714 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 20 client.15714 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa148460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 4 client.15714 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624fa0a1bf0 con 0x5624fa146070 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.335+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f769affd640 10 client.15714 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:52.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa148460 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67ad2e8316de2d75 server_cookie=3933e07a14de0f3d in_seq=0 out_seq=0 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f769affd640 1 -- 172.21.15.107:0/2358488442 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624fa0a1bf0 con 0x5624fa146070 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f769affd640 10 client.15714 handle_command_reply: tid=0 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f76c4e0e740 1 client.15714 shutdown 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 msgr2=0x5624fa13abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.336+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76b002aa00 tx=0x7f76b00046c0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 shutdown_connections 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa148460 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f769affd640 0 client.15714 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130c30 0x5624fa13abb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/2358488442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130240 0x5624fa13a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 >> 172.21.15.107:0/2358488442 conn(0x5624fa12b8b0 msgr2=0x5624fa12aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 shutdown_connections 2024-04-24T23:45:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 1 -- 172.21.15.107:0/2358488442 wait complete. 2024-04-24T23:45:52.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.337+0000 7f76c4e0e740 20 client.15714 trim_cache size 0 max 16384 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 Processor -- start 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 -- start start 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa141b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 0x5624fa12eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1aa570 con 0x5624fa130240 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1ac0b0 con 0x5624fa130c30 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.354+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1aac40 con 0x5624fa135da0 2024-04-24T23:45:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51270/0 (socket says 172.21.15.107:51270) 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1812166898 learned_addr learned my addr 172.21.15.107:0/1812166898 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76baffd640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bbfff640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 0x5624fa12eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1812166898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 msgr2=0x5624fa12eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 0x5624fa12eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1812166898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 msgr2=0x5624fa141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1812166898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624fa1487d0 con 0x5624fa130240 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bbfff640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 0x5624fa12eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00be60 tx=0x7f76ac0d4630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be76cc2dd8fd62 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76a17fa640 1 -- 172.21.15.107:0/1812166898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0d4070 con 0x5624fa130240 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76a17fa640 1 -- 172.21.15.107:0/1812166898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ac09ad20 con 0x5624fa130240 2024-04-24T23:45:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.355+0000 7f76a17fa640 1 -- 172.21.15.107:0/1812166898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac01e040 con 0x5624fa130240 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1812166898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 msgr2=0x5624fa12e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00be60 tx=0x7f76ac0d4630 comp rx=0 tx=0).stop 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1812166898 shutdown_connections 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa135da0 0x5624fa12eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa130c30 0x5624fa141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1812166898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa12e630 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1812166898 >> 172.21.15.107:0/1812166898 conn(0x5624fa12b8b0 msgr2=0x5624fa1320d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1812166898 shutdown_connections 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1812166898 wait complete. 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.356+0000 7f76c4e0e740 1 -- start start 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 Processor -- start 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130c30 0x5624fa13f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1ab9e0 con 0x5624fa130240 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1ab310 con 0x5624fa135da0 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1a8550 con 0x5624fa130c30 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76c4e0e740 10 client.0 _ll_register_callbacks cb 0x7f76ac0d4500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51280/0 (socket says 172.21.15.107:51280) 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1219014687 learned_addr learned my addr 172.21.15.107:0/1219014687 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1219014687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130c30 msgr2=0x5624fa13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130c30 0x5624fa13f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 msgr2=0x5624fa1407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bbfff640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa1407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 -- 172.21.15.107:0/1219014687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624fa140dd0 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f76bb7fe640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f76ac000c00 tx=0x7f76ac0a3990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ceba8f093939efd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.357+0000 7f769affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0b7070 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f76ac097390 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ac09ad20 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76ac01e070 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 10 client.15720 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5624fa1ac580 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ac09ad20 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f76ac079cb0 con 0x5624fa130240 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 10 client.15720 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 20 client.15720 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa1aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 4 client.15720 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.358+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624fa09bf40 con 0x5624fa146070 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76baffd640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa1aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f769affd640 10 client.15720 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76baffd640 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa1aef50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c52618317b73bb0 server_cookie=23d4b294b2de447a in_seq=0 out_seq=0 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f769affd640 1 -- 172.21.15.107:0/1219014687 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5624fa09bf40 con 0x5624fa146070 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f769affd640 10 client.15720 handle_command_reply: tid=0 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 client.15720 shutdown 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 msgr2=0x5624fa13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f76ac000c00 tx=0x7f76ac0a3990 comp rx=0 tx=0).stop 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 shutdown_connections 2024-04-24T23:45:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5624fa146070 0x5624fa1aef50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5624fa135da0 0x5624fa1407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f769affd640 0 client.15720 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5624fa130c30 0x5624fa13f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 --2- 172.21.15.107:0/1219014687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5624fa130240 0x5624fa13f290 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.359+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 >> 172.21.15.107:0/1219014687 conn(0x5624fa12b8b0 msgr2=0x5624fa12a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 shutdown_connections 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c4e0e740 1 -- 172.21.15.107:0/1219014687 wait complete. 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c4e0e740 20 client.15720 trim_cache size 0 max 16384 2024-04-24T23:45:52.366 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-04-24T23:45:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-04-24T23:45:52.367 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 msgr2=0x7f76a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 0x7f76a40a5050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x5624fa131490 tx=0x7f76b4004a40 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 msgr2=0x7f76bc197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00b4a0 tx=0x7f76ac00b960 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 shutdown_connections 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f76a40a2b90 0x7f76a40a5050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f76bc107460 0x7f76bc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc104b30 0x7f76bc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 --2- 172.21.15.107:0/3135892598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76bc102200 0x7f76bc197b20 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 >> 172.21.15.107:0/3135892598 conn(0x7f76bc0fbbf0 msgr2=0x7f76bc102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 shutdown_connections 2024-04-24T23:45:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.360+0000 7f76c202c640 1 -- 172.21.15.107:0/3135892598 wait complete. 2024-04-24T23:45:52.371 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T23:45:52.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 -- 172.21.15.107:0/3876138668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10100680 msgr2=0x7fda10100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 --2- 172.21.15.107:0/3876138668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10100680 0x7fda10100aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009900 tx=0x7fda0c02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 -- 172.21.15.107:0/3876138668 shutdown_connections 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 --2- 172.21.15.107:0/3876138668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda10108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 --2- 172.21.15.107:0/3876138668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10101870 0x7fda10101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 --2- 172.21.15.107:0/3876138668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10100680 0x7fda10100aa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 -- 172.21.15.107:0/3876138668 >> 172.21.15.107:0/3876138668 conn(0x7fda100fbbf0 msgr2=0x7fda100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 -- 172.21.15.107:0/3876138668 shutdown_connections 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.473+0000 7fda172a5640 1 -- 172.21.15.107:0/3876138668 wait complete. 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 Processor -- start 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 -- start start 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 0x7fda10197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda1019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda1010c2d0 con 0x7fda10101870 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fda1010c150 con 0x7fda10100680 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda172a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fda1010c450 con 0x7fda10102230 2024-04-24T23:45:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51300/0 (socket says 172.21.15.107:51300) 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 -- 172.21.15.107:0/3951264616 learned_addr learned my addr 172.21.15.107:0/3951264616 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda15842640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda1019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda15041640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 0x7fda10197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 -- 172.21.15.107:0/3951264616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 msgr2=0x7fda1019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda1019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 -- 172.21.15.107:0/3951264616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 msgr2=0x7fda10197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 0x7fda10197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 -- 172.21.15.107:0/3951264616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1019f680 con 0x7fda10101870 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda15842640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda1019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda15041640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 0x7fda10197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda14840640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e9f0 tx=0x7fda0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.474+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0000cd80 con 0x7fda10101870 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.475+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1019f970 con 0x7fda10101870 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.475+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda00010910 con 0x7fda10101870 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.475+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda000188f0 con 0x7fda10101870 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.475+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1019ff10 con 0x7fda10101870 2024-04-24T23:45:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.476+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda101a0140 con 0x7fda10101870 2024-04-24T23:45:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.477+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fda00010ac0 con 0x7fda10101870 2024-04-24T23:45:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.477+0000 7fda067fc640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 0x7fd9fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.477+0000 7fda15041640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 0x7fd9fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.477+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fda000cf900 con 0x7fda10101870 2024-04-24T23:45:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.478+0000 7fda15041640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 0x7fd9fc0a5050 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.482+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda0009c490 con 0x7fda10101870 2024-04-24T23:45:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.707+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda101a03b0 con 0x7fda10101870 2024-04-24T23:45:52.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.708+0000 7fda067fc640 1 -- 172.21.15.107:0/3951264616 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7fda0009bd90 con 0x7fda10101870 2024-04-24T23:45:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:45:52.708 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:45:52.708 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 msgr2=0x7fd9fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 0x7fd9fc0a5050 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c031400 comp rx=0 tx=0).stop 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 msgr2=0x7fda10197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e9f0 tx=0x7fda0000eeb0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 shutdown_connections 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9fc0a2b90 0x7fd9fc0a5050 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda10102230 0x7fda1019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda10101870 0x7fda10197fa0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 --2- 172.21.15.107:0/3951264616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda10100680 0x7fda10197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 >> 172.21.15.107:0/3951264616 conn(0x7fda100fbbf0 msgr2=0x7fda100fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 shutdown_connections 2024-04-24T23:45:52.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.710+0000 7fda172a5640 1 -- 172.21.15.107:0/3951264616 wait complete. 2024-04-24T23:45:52.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-24T23:45:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 -- 172.21.15.107:0/2082797774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 msgr2=0x7fa80c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 --2- 172.21.15.107:0/2082797774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c108ac0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa804009ca0 tx=0x7fa80402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 -- 172.21.15.107:0/2082797774 shutdown_connections 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 --2- 172.21.15.107:0/2082797774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c108ac0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 --2- 172.21.15.107:0/2082797774 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 --2- 172.21.15.107:0/2082797774 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 0x7fa80c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.824+0000 7fa811c38640 1 -- 172.21.15.107:0/2082797774 >> 172.21.15.107:0/2082797774 conn(0x7fa80c0fbbf0 msgr2=0x7fa80c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- 172.21.15.107:0/2082797774 shutdown_connections 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- 172.21.15.107:0/2082797774 wait complete. 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 Processor -- start 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- start start 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 0x7fa80c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10c360 con 0x7fa80c102230 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10c1e0 con 0x7fa80c100680 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.825+0000 7fa811c38640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10c4e0 con 0x7fa80c101870 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51318/0 (socket says 172.21.15.107:51318) 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 -- 172.21.15.107:0/1099282901 learned_addr learned my addr 172.21.15.107:0/1099282901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 0x7fa80c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80affd640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 -- 172.21.15.107:0/1099282901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 msgr2=0x7fa80c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 -- 172.21.15.107:0/1099282901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 msgr2=0x7fa80c069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 0x7fa80c069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 -- 172.21.15.107:0/1099282901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c19f730 con 0x7fa80c102230 2024-04-24T23:45:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80affd640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa80bfff640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa804005e30 tx=0x7fa804031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa808ff9640 1 -- 172.21.15.107:0/1099282901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804003c30 con 0x7fa80c102230 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80c19f9c0 con 0x7fa80c102230 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa808ff9640 1 -- 172.21.15.107:0/1099282901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa804003dd0 con 0x7fa80c102230 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa808ff9640 1 -- 172.21.15.107:0/1099282901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80402fde0 con 0x7fa80c102230 2024-04-24T23:45:52.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.826+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa80c19ff90 con 0x7fa80c102230 2024-04-24T23:45:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 Processor -- start 2024-04-24T23:45:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa808ff9640 1 -- 172.21.15.107:0/1099282901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa804004050 con 0x7fa80c102230 2024-04-24T23:45:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 -- start start 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fb230 0x55fda52fb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda5303220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fda525f670 con 0x55fda52fbc20 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fda5207970 con 0x55fda5300d90 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa814a29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fda525b630 con 0x55fda52fb230 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa80b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa80b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51332/0 (socket says 172.21.15.107:51332) 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa80b7fe640 1 -- 172.21.15.107:0/28352208 learned_addr learned my addr 172.21.15.107:0/28352208 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa80affd640 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda5303220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa808ff9640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 0x7fa7f40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.829+0000 7fa80affd640 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fb230 0x55fda52fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 -- 172.21.15.107:0/28352208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fb230 msgr2=0x55fda52fb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fb230 0x55fda52fb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 -- 172.21.15.107:0/28352208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 msgr2=0x55fda5303220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda5303220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 -- 172.21.15.107:0/28352208 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda522b400 con 0x55fda52fbc20 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80bfff640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 0x7fa7f40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa808ff9640 1 -- 172.21.15.107:0/1099282901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa8040f0930 con 0x7fa80c102230 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa80000bcb0 tx=0x7fa8000107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbb3b217b619e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa7f1ffb640 1 -- 172.21.15.107:0/28352208 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800004020 con 0x55fda52fbc20 2024-04-24T23:45:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa7f1ffb640 1 -- 172.21.15.107:0/28352208 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa800005500 con 0x55fda52fbc20 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.830+0000 7fa7f1ffb640 1 -- 172.21.15.107:0/28352208 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800004de0 con 0x55fda52fbc20 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- 172.21.15.107:0/28352208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 msgr2=0x55fda5300720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa80bfff640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 0x7fa7f40a5050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8040bcbe0 tx=0x7fa8040b23c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa80000bcb0 tx=0x7fa8000107a0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- 172.21.15.107:0/28352208 shutdown_connections 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda5303220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fbc20 0x55fda5300720 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 --2- 172.21.15.107:0/28352208 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fb230 0x55fda52fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- 172.21.15.107:0/28352208 >> 172.21.15.107:0/28352208 conn(0x55fda52f68a0 msgr2=0x55fda52f6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- 172.21.15.107:0/28352208 shutdown_connections 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- 172.21.15.107:0/28352208 wait complete. 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.831+0000 7fa814a29740 1 -- start start 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 Processor -- start 2024-04-24T23:45:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda5305870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda530c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fda5303d40 con 0x55fda52fb230 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fda521b050 con 0x55fda5300d90 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fda5303ec0 con 0x55fda52fbc20 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa814a29740 10 client.0 _ll_register_callbacks cb 0x55fda4f72be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51336/0 (socket says 172.21.15.107:51336) 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 -- 172.21.15.107:0/1036706308 learned_addr learned my addr 172.21.15.107:0/1036706308 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda5305870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda530c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 -- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 msgr2=0x55fda5305870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda5305870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 -- 172.21.15.107:0/1036706308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 msgr2=0x55fda530c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda530c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 -- 172.21.15.107:0/1036706308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda51ecdc0 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa80affd640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00cf30 tx=0x7fa7fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2e57785c2a5e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc01c070 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.832+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7fc0183f0 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa800005500 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7fc00f040 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 10 client.15738 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fda4f733f0 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa800005500 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa7fc015040 con 0x55fda52fb230 2024-04-24T23:45:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 10 client.15738 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 10 client.15738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 20 client.15738 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5311170 0x55fda5313610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 4 client.15738 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fda526cbb0 con 0x55fda5311170 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.833+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5311170 0x55fda5313610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.834+0000 7fa7eaffd640 10 client.15738 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.834+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5311170 0x55fda5313610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9c5fce7dba5e965e server_cookie=63047043e4bcabd2 in_seq=0 out_seq=0 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.834+0000 7fa7eaffd640 1 -- 172.21.15.107:0/1036706308 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fda526cbb0 con 0x55fda5311170 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.834+0000 7fa7eaffd640 10 client.15738 handle_command_reply: tid=0 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.834+0000 7fa814a29740 1 client.15738 shutdown 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 msgr2=0x55fda5305330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00cf30 tx=0x7fa7fc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 shutdown_connections 2024-04-24T23:45:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5311170 0x55fda5313610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda530c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda5305870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa7eaffd640 0 client.15738 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 --2- 172.21.15.107:0/1036706308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda5305330 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 >> 172.21.15.107:0/1036706308 conn(0x55fda52f68a0 msgr2=0x55fda52f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 shutdown_connections 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 1 -- 172.21.15.107:0/1036706308 wait complete. 2024-04-24T23:45:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.835+0000 7fa814a29740 20 client.15738 trim_cache size 0 max 16384 2024-04-24T23:45:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 Processor -- start 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 -- start start 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda530c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda52f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1aa5b0 con 0x55fda52fb230 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1ac0f0 con 0x55fda5300d90 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa814a29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1aac80 con 0x55fda52fbc20 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51340/0 (socket says 172.21.15.107:51340) 2024-04-24T23:45:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.848+0000 7fa80affd640 1 -- 172.21.15.107:0/2790631052 learned_addr learned my addr 172.21.15.107:0/2790631052 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda530c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda52f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 -- 172.21.15.107:0/2790631052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 msgr2=0x55fda530c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda530c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 -- 172.21.15.107:0/2790631052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 msgr2=0x55fda52f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda52f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 -- 172.21.15.107:0/2790631052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda503d7d0 con 0x55fda52fb230 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa80affd640 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc004690 tx=0x7fa7fc017f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab00c2e4d1f60025 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa7f17fa640 1 -- 172.21.15.107:0/2790631052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc01c070 con 0x55fda52fb230 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa7f17fa640 1 -- 172.21.15.107:0/2790631052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc014b50 con 0x55fda52fb230 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.849+0000 7fa7f17fa640 1 -- 172.21.15.107:0/2790631052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc018860 con 0x55fda52fb230 2024-04-24T23:45:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- 172.21.15.107:0/2790631052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 msgr2=0x55fda53057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc004690 tx=0x7fa7fc017f60 comp rx=0 tx=0).stop 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- 172.21.15.107:0/2790631052 shutdown_connections 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda5300d90 0x55fda52f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda52fbc20 0x55fda530c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- 172.21.15.107:0/2790631052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda53057d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- 172.21.15.107:0/2790631052 >> 172.21.15.107:0/2790631052 conn(0x55fda52f68a0 msgr2=0x55fda52f52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- 172.21.15.107:0/2790631052 shutdown_connections 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- 172.21.15.107:0/2790631052 wait complete. 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 -- start start 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 Processor -- start 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.850+0000 7fa814a29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda52fbc20 0x55fda530a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda5300d90 0x55fda530b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1aba20 con 0x55fda52fb230 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1ab350 con 0x55fda52fbc20 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c1a8590 con 0x55fda5300d90 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 10 client.0 _ll_register_callbacks cb 0x7fa7fc00cb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51356/0 (socket says 172.21.15.107:51356) 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 -- 172.21.15.107:0/3426202821 learned_addr learned my addr 172.21.15.107:0/3426202821 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda52fbc20 0x55fda530a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda5300d90 0x55fda530b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 -- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda5300d90 msgr2=0x55fda530b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda5300d90 0x55fda530b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 -- 172.21.15.107:0/3426202821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda52fbc20 msgr2=0x55fda530a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda52fbc20 0x55fda530a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 -- 172.21.15.107:0/3426202821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda530bd50 con 0x55fda52fb230 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa80affd640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc018bd0 tx=0x7fa7fc017d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f59110e237bc8b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc03c060 con 0x55fda52fb230 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7fc0059c0 con 0x55fda52fb230 2024-04-24T23:45:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7fc014b50 con 0x55fda52fb230 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7fc01c040 con 0x55fda52fb230 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 10 client.15744 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.851+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fda53779f0 con 0x55fda52fb230 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc014b50 con 0x55fda52fb230 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa7fc02d670 con 0x55fda52fb230 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 10 client.15744 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 10 client.15744 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 20 client.15744 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5378340 0x55fda537a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 4 client.15744 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fda5266f00 con 0x55fda5378340 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5378340 0x55fda537a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa7eaffd640 10 client.15744 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.852+0000 7fa80b7fe640 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5378340 0x55fda537a760 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea5120523f011030 server_cookie=370d9d97e19511b5 in_seq=0 out_seq=0 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa7eaffd640 1 -- 172.21.15.107:0/3426202821 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fda5266f00 con 0x55fda5378340 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa7eaffd640 10 client.15744 handle_command_reply: tid=0 2024-04-24T23:45:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 client.15744 shutdown 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 msgr2=0x55fda530a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc018bd0 tx=0x7fa7fc017d70 comp rx=0 tx=0).stop 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 shutdown_connections 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55fda5378340 0x55fda537a760 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fda5300d90 0x55fda530b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fda52fbc20 0x55fda530a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa7eaffd640 0 client.15744 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 --2- 172.21.15.107:0/3426202821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda52fb230 0x55fda530a210 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 >> 172.21.15.107:0/3426202821 conn(0x55fda52f68a0 msgr2=0x55fda5305650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 shutdown_connections 2024-04-24T23:45:52.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 1 -- 172.21.15.107:0/3426202821 wait complete. 2024-04-24T23:45:52.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa814a29740 20 client.15744 trim_cache size 0 max 16384 2024-04-24T23:45:52.863 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:45:52.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 msgr2=0x7fa7f40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 0x7fa7f40a5050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8040bcbe0 tx=0x7fa8040b23c0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 msgr2=0x7fa80c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.853+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa804005e30 tx=0x7fa804031ee0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 shutdown_connections 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa7f40a2b90 0x7fa7f40a5050 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa80c102230 0x7fa80c19ee70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa80c101870 0x7fa80c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 --2- 172.21.15.107:0/1099282901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa80c100680 0x7fa80c069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 >> 172.21.15.107:0/1099282901 conn(0x7fa80c0fbbf0 msgr2=0x7fa80c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 shutdown_connections 2024-04-24T23:45:52.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.854+0000 7fa811c38640 1 -- 172.21.15.107:0/1099282901 wait complete. 2024-04-24T23:45:52.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 -- 172.21.15.107:0/661644734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508103340 msgr2=0x7f9508109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 --2- 172.21.15.107:0/661644734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508103340 0x7f9508109bd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9500009ca0 tx=0x7f950002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 -- 172.21.15.107:0/661644734 shutdown_connections 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 --2- 172.21.15.107:0/661644734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508103340 0x7f9508109bd0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 --2- 172.21.15.107:0/661644734 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 --2- 172.21.15.107:0/661644734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508101790 0x7f9508101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 -- 172.21.15.107:0/661644734 >> 172.21.15.107:0/661644734 conn(0x7f95080fcd20 msgr2=0x7f95080ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 -- 172.21.15.107:0/661644734 shutdown_connections 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.966+0000 7f950e54f640 1 -- 172.21.15.107:0/661644734 wait complete. 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 Processor -- start 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 -- start start 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508103340 0x7f950819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d3f0 con 0x7f9508101790 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d270 con 0x7f9508103340 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f950e54f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f950810d570 con 0x7f9508102980 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f9507fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f9507fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51382/0 (socket says 172.21.15.107:51382) 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f9507fff640 1 -- 172.21.15.107:0/1935970287 learned_addr learned my addr 172.21.15.107:0/1935970287 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.967+0000 7f95077fe640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 -- 172.21.15.107:0/1935970287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 msgr2=0x7f9508197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f950caec640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508103340 0x7f950819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 -- 172.21.15.107:0/1935970287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508103340 msgr2=0x7f950819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508103340 0x7f950819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 -- 172.21.15.107:0/1935970287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819f650 con 0x7f9508101790 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f95077fe640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f9507fff640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d770 tx=0x7f94f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f95057fa640 1 -- 172.21.15.107:0/1935970287 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8017070 con 0x7f9508101790 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f95057fa640 1 -- 172.21.15.107:0/1935970287 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f8009c30 con 0x7f9508101790 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819f940 con 0x7f9508101790 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f95057fa640 1 -- 172.21.15.107:0/1935970287 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8003dd0 con 0x7f9508101790 2024-04-24T23:45:52.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.968+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950819fe90 con 0x7f9508101790 2024-04-24T23:45:52.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.970+0000 7f95057fa640 1 -- 172.21.15.107:0/1935970287 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f94f8003f70 con 0x7f9508101790 2024-04-24T23:45:52.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f95057fa640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 0x7f94f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f95077fe640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 0x7f94f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f95057fa640 1 -- 172.21.15.107:0/1935970287 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f94f80ceea0 con 0x7f9508101790 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f95077fe640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 0x7f94f00a5050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0058c0 tx=0x7f94fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f9511256740 1 Processor -- start 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.971+0000 7f9511256740 1 -- start start 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d11650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d16720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5627c9c75670 con 0x5627c9d16d90 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5627c9c1d970 con 0x5627c9d11c20 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9511256740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5627c9c71630 con 0x5627c9d11230 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51384/0 (socket says 172.21.15.107:51384) 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 -- 172.21.15.107:0/1034446209 learned_addr learned my addr 172.21.15.107:0/1034446209 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f95077fe640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d16720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f9507fff640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d11650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 -- 172.21.15.107:0/1034446209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 msgr2=0x5627c9d11650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d11650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 -- 172.21.15.107:0/1034446209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 msgr2=0x5627c9d16720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d16720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 -- 172.21.15.107:0/1034446209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c9c41400 con 0x5627c9d16d90 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f95077fe640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d16720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f950caec640 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f950002ade0 tx=0x7f9500005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccbefcc8eab5294 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f94ee7fc640 1 -- 172.21.15.107:0/1034446209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500031450 con 0x5627c9d16d90 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f94ee7fc640 1 -- 172.21.15.107:0/1034446209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500036070 con 0x5627c9d16d90 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.972+0000 7f94ee7fc640 1 -- 172.21.15.107:0/1034446209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000043c0 con 0x5627c9d16d90 2024-04-24T23:45:52.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- 172.21.15.107:0/1034446209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 msgr2=0x5627c9d19220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f950002ade0 tx=0x7f9500005f70 comp rx=0 tx=0).stop 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- 172.21.15.107:0/1034446209 shutdown_connections 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d19220 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d16720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 --2- 172.21.15.107:0/1034446209 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d11650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- 172.21.15.107:0/1034446209 >> 172.21.15.107:0/1034446209 conn(0x5627c9d0c8a0 msgr2=0x5627c9d0cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- 172.21.15.107:0/1034446209 shutdown_connections 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- 172.21.15.107:0/1034446209 wait complete. 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.973+0000 7f9511256740 1 -- start start 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 Processor -- start 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11230 0x5627c9d1b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11c20 0x5627c9d1b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5627c9d19ce0 con 0x5627c9d16d90 2024-04-24T23:45:52.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5627c9c31050 con 0x5627c9d11230 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5627c9d19e60 con 0x5627c9d11c20 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9511256740 10 client.0 _ll_register_callbacks cb 0x5627c9988be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51396/0 (socket says 172.21.15.107:51396) 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f950caec640 1 -- 172.21.15.107:0/1049738242 learned_addr learned my addr 172.21.15.107:0/1049738242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f9507fff640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11230 0x5627c9d1b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f950caec640 1 -- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11c20 msgr2=0x5627c9d1b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.974+0000 7f95077fe640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11c20 0x5627c9d1b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f950caec640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11c20 0x5627c9d1b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f950caec640 1 -- 172.21.15.107:0/1049738242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11230 msgr2=0x5627c9d1b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f950caec640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11230 0x5627c9d1b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f950caec640 1 -- 172.21.15.107:0/1049738242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c9c02dc0 con 0x5627c9d16d90 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f950caec640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f950002a900 tx=0x7f9500032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226b35584f4a1a0a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000316e0 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9500003970 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9500036070 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f950005b050 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f9511256740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627c99893f0 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500036070 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.975+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f950002f730 con 0x5627c9d16d90 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 10 client.15756 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 20 client.15756 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d26ce0 0x5627c9d290d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 4 client.15756 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627c9c82bb0 con 0x5627c9d26ce0 2024-04-24T23:45:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9507fff640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d26ce0 0x5627c9d290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f9507fff640 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d26ce0 0x5627c9d290d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3b8a0318d57e847 server_cookie=460c82bea6869504 in_seq=0 out_seq=0 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.976+0000 7f94e77fe640 10 client.15756 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f94e77fe640 1 -- 172.21.15.107:0/1049738242 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f94f8080ae0 con 0x5627c9d26ce0 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f94e77fe640 10 client.15756 handle_command_reply: tid=0 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 client.15756 shutdown 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 msgr2=0x5627c9d227f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f950002a900 tx=0x7f9500032eb0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 shutdown_connections 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d26ce0 0x5627c9d290d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d227f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11c20 0x5627c9d1b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f94e77fe640 0 client.15756 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 --2- 172.21.15.107:0/1049738242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11230 0x5627c9d1b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 >> 172.21.15.107:0/1049738242 conn(0x5627c9d0c8a0 msgr2=0x5627c9d0bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 shutdown_connections 2024-04-24T23:45:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 1 -- 172.21.15.107:0/1049738242 wait complete. 2024-04-24T23:45:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.977+0000 7f9511256740 20 client.15756 trim_cache size 0 max 16384 2024-04-24T23:45:52.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 Processor -- start 2024-04-24T23:45:52.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 -- start start 2024-04-24T23:45:52.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d12dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d227b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95081aa430 con 0x5627c9d16d90 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f95081abf70 con 0x5627c9d11c20 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f9511256740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f95081aab00 con 0x5627c9d11230 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51412/0 (socket says 172.21.15.107:51412) 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 -- 172.21.15.107:0/2146718549 learned_addr learned my addr 172.21.15.107:0/2146718549 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 -- 172.21.15.107:0/2146718549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 msgr2=0x5627c9d12dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d12dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f95077fe640 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d227b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 -- 172.21.15.107:0/2146718549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 msgr2=0x5627c9d227b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d227b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 -- 172.21.15.107:0/2146718549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c9a537d0 con 0x5627c9d16d90 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.991+0000 7f950caec640 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f950002ade0 tx=0x7f95000021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cec30cfd94c02c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f94edffb640 1 -- 172.21.15.107:0/2146718549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000305c0 con 0x5627c9d16d90 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f94edffb640 1 -- 172.21.15.107:0/2146718549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500036070 con 0x5627c9d16d90 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f94edffb640 1 -- 172.21.15.107:0/2146718549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500003bd0 con 0x5627c9d16d90 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 -- 172.21.15.107:0/2146718549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 msgr2=0x5627c9d2d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f950002ade0 tx=0x7f95000021d0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 -- 172.21.15.107:0/2146718549 shutdown_connections 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d16d90 0x5627c9d2d970 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d11c20 0x5627c9d227b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 --2- 172.21.15.107:0/2146718549 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d12dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 -- 172.21.15.107:0/2146718549 >> 172.21.15.107:0/2146718549 conn(0x5627c9d0c8a0 msgr2=0x5627c9d126c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 -- 172.21.15.107:0/2146718549 shutdown_connections 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.992+0000 7f9511256740 1 -- 172.21.15.107:0/2146718549 wait complete. 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 -- start start 2024-04-24T23:45:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 Processor -- start 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d22500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d16d90 0x5627c9d21050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95081ab8a0 con 0x5627c9d11c20 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f95081ab1d0 con 0x5627c9d16d90 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a8410 con 0x5627c9d11230 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9511256740 10 client.0 _ll_register_callbacks cb 0x7f9500009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51420/0 (socket says 172.21.15.107:51420) 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 -- 172.21.15.107:0/2937733119 learned_addr learned my addr 172.21.15.107:0/2937733119 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f950caec640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d16d90 0x5627c9d21050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f9507fff640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d22500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 -- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 msgr2=0x5627c9d22500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d22500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 -- 172.21.15.107:0/2937733119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d16d90 msgr2=0x5627c9d21050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d16d90 0x5627c9d21050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.993+0000 7f95077fe640 1 -- 172.21.15.107:0/2937733119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c9d216c0 con 0x5627c9d11c20 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f95077fe640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9508198f40 tx=0x7f94fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a1d7bfc6310e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc020850 con 0x5627c9d11c20 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9500036070 con 0x5627c9d11c20 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f94fc0209f0 con 0x5627c9d11c20 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94fc032050 con 0x5627c9d11c20 2024-04-24T23:45:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 10 client.15762 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5627c9d8e1c0 con 0x5627c9d11c20 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500036070 con 0x5627c9d11c20 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f94fc01f440 con 0x5627c9d11c20 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 10 client.15762 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 10 client.15762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 20 client.15762 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d8eb20 0x5627c9d90f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 4 client.15762 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.994+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627c9c7cf00 con 0x5627c9d8eb20 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9507fff640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d8eb20 0x5627c9d90f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9507fff640 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d8eb20 0x5627c9d90f40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b9af317d52fa526f server_cookie=a4fd91b272dd2b87 in_seq=0 out_seq=0 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f94e77fe640 10 client.15762 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f94e77fe640 1 -- 172.21.15.107:0/2937733119 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627c9c7cf00 con 0x5627c9d8eb20 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f94e77fe640 10 client.15762 handle_command_reply: tid=0 2024-04-24T23:45:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 client.15762 shutdown 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 msgr2=0x5627c9d20b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9508198f40 tx=0x7f94fc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 shutdown_connections 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5627c9d8eb20 0x5627c9d90f40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5627c9d16d90 0x5627c9d21050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627c9d11c20 0x5627c9d20b10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 --2- 172.21.15.107:0/2937733119 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5627c9d11230 0x5627c9d22500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f94e77fe640 0 client.15762 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 >> 172.21.15.107:0/2937733119 conn(0x5627c9d0c8a0 msgr2=0x5627c9d12f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 shutdown_connections 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.995+0000 7f9511256740 1 -- 172.21.15.107:0/2937733119 wait complete. 2024-04-24T23:45:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f9511256740 20 client.15762 trim_cache size 0 max 16384 2024-04-24T23:45:53.002 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 msgr2=0x7f94f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 0x7f94f00a5050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0058c0 tx=0x7f94fc005850 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 msgr2=0x7f9508197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d770 tx=0x7f94f800dc30 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 shutdown_connections 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f94f00a2b90 0x7f94f00a5050 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9508103340 0x7f950819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9508102980 0x7f9508197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 --2- 172.21.15.107:0/1935970287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9508101790 0x7f9508197a30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 >> 172.21.15.107:0/1935970287 conn(0x7f95080fcd20 msgr2=0x7f9508105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 shutdown_connections 2024-04-24T23:45:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:52.996+0000 7f950e54f640 1 -- 172.21.15.107:0/1935970287 wait complete. 2024-04-24T23:45:53.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 -- 172.21.15.107:0/3206016824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c101870 msgr2=0x7f0a7c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3206016824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c101870 0x7f0a7c101cf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009900 tx=0x7f0a6c02a410 comp rx=0 tx=0).stop 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 -- 172.21.15.107:0/3206016824 shutdown_connections 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3206016824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c102230 0x7f0a7c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3206016824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c101870 0x7f0a7c101cf0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3206016824 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c100680 0x7f0a7c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 -- 172.21.15.107:0/3206016824 >> 172.21.15.107:0/3206016824 conn(0x7f0a7c0fbbf0 msgr2=0x7f0a7c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 -- 172.21.15.107:0/3206016824 shutdown_connections 2024-04-24T23:45:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 -- 172.21.15.107:0/3206016824 wait complete. 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.109+0000 7f0a83d16640 1 Processor -- start 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 -- start start 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c101870 0x7f0a7c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c102230 0x7f0a7c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c10c380 con 0x7f0a7c100680 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c10c200 con 0x7f0a7c101870 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c10c500 con 0x7f0a7c102230 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51440/0 (socket says 172.21.15.107:51440) 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3661811627 learned_addr learned my addr 172.21.15.107:0/3661811627 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a822b3640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c102230 0x7f0a7c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a812b1640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c101870 0x7f0a7c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3661811627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c102230 msgr2=0x7f0a7c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c102230 0x7f0a7c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3661811627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c101870 msgr2=0x7f0a7c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c101870 0x7f0a7c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3661811627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7c103a20 con 0x7f0a7c100680 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c970 tx=0x7f0a7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a72ffd640 1 -- 172.21.15.107:0/3661811627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78004220 con 0x7f0a7c100680 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a72ffd640 1 -- 172.21.15.107:0/3661811627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a780043c0 con 0x7f0a7c100680 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7c1a1c40 con 0x7f0a7c100680 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.110+0000 7f0a72ffd640 1 -- 172.21.15.107:0/3661811627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78004a00 con 0x7f0a7c100680 2024-04-24T23:45:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.111+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7c1a2160 con 0x7f0a7c100680 2024-04-24T23:45:53.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.113+0000 7f0a86a1a740 1 Processor -- start 2024-04-24T23:45:53.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.113+0000 7f0a72ffd640 1 -- 172.21.15.107:0/3661811627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0a780079b0 con 0x7f0a7c100680 2024-04-24T23:45:53.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.113+0000 7f0a86a1a740 1 -- start start 2024-04-24T23:45:53.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b86c20 0x559132b8b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b8bd90 0x559132b8e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559132aea670 con 0x559132b86230 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559132a92970 con 0x559132b8bd90 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559132ae6630 con 0x559132b86c20 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51456/0 (socket says 172.21.15.107:51456) 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 -- 172.21.15.107:0/2216387565 learned_addr learned my addr 172.21.15.107:0/2216387565 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b86c20 0x559132b8b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a812b1640 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b8bd90 0x559132b8e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a72ffd640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 0x7f0a680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 -- 172.21.15.107:0/2216387565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b86c20 msgr2=0x559132b8b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b86c20 0x559132b8b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 -- 172.21.15.107:0/2216387565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b8bd90 msgr2=0x559132b8e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b8bd90 0x559132b8e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 -- 172.21.15.107:0/2216387565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559132ab6400 con 0x559132b86230 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 0x7f0a680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a72ffd640 1 -- 172.21.15.107:0/3661811627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0a780cf190 con 0x7f0a7c100680 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.114+0000 7f0a822b3640 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400a5d0 tx=0x7f0a7400aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4153ff00de06a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a822b3640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 0x7f0a680a50d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0a74020000 tx=0x7f0a74021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a67fff640 1 -- 172.21.15.107:0/2216387565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7401a040 con 0x559132b86230 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a67fff640 1 -- 172.21.15.107:0/2216387565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a74021ba0 con 0x559132b86230 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a67fff640 1 -- 172.21.15.107:0/2216387565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7401f350 con 0x559132b86230 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2216387565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 msgr2=0x559132b86650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.115+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400a5d0 tx=0x7f0a7400aa90 comp rx=0 tx=0).stop 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2216387565 shutdown_connections 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b8bd90 0x559132b8e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b86c20 0x559132b8b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2216387565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b86650 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2216387565 >> 172.21.15.107:0/2216387565 conn(0x559132b818a0 msgr2=0x559132b81ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2216387565 shutdown_connections 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2216387565 wait complete. 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- start start 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 Processor -- start 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b978e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559132b8ed40 con 0x559132b86230 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559132aa6050 con 0x559132b86c20 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559132b8eec0 con 0x559132b8bd90 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a86a1a740 10 client.0 _ll_register_callbacks cb 0x5591327fdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.116+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51462/0 (socket says 172.21.15.107:51462) 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 -- 172.21.15.107:0/1920009690 learned_addr learned my addr 172.21.15.107:0/1920009690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a812b1640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a822b3640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 -- 172.21.15.107:0/1920009690 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 msgr2=0x559132b978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 -- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 msgr2=0x559132b908a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b908a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 -- 172.21.15.107:0/1920009690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559132a77dc0 con 0x559132b86230 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a78007590 tx=0x7f0a7800a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a5914ef13b3aeb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7801d040 con 0x559132b86230 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0a78084bb0 con 0x559132b86230 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a74021ba0 con 0x559132b86230 2024-04-24T23:45:53.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a7809bd20 con 0x559132b86230 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a86a1a740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591327fe3f0 con 0x559132b86230 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.117+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a74021ba0 con 0x559132b86230 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0a7809aa70 con 0x559132b86230 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 10 client.15774 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 20 client.15774 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132b9c210 0x559132b9e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 4 client.15774 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559132af7bb0 con 0x559132b9c210 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a812b1640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132b9c210 0x559132b9e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a657fa640 10 client.15774 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.118+0000 7f0a812b1640 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132b9c210 0x559132b9e6a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d935fee76b3339e server_cookie=2601f3927c350a22 in_seq=0 out_seq=0 2024-04-24T23:45:53.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a657fa640 1 -- 172.21.15.107:0/1920009690 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559132af7bb0 con 0x559132b9c210 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a657fa640 10 client.15774 handle_command_reply: tid=0 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a86a1a740 1 client.15774 shutdown 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 msgr2=0x559132b90360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a78007590 tx=0x7f0a7800a4a0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 shutdown_connections 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.119+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132b9c210 0x559132b9e6a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/1920009690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b90360 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a657fa640 0 client.15774 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 >> 172.21.15.107:0/1920009690 conn(0x559132b818a0 msgr2=0x559132b80b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 shutdown_connections 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 1 -- 172.21.15.107:0/1920009690 wait complete. 2024-04-24T23:45:53.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.120+0000 7f0a86a1a740 20 client.15774 trim_cache size 0 max 16384 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 Processor -- start 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 -- start start 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 0x559132b84580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b84ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1ac800 con 0x559132b86c20 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1ae340 con 0x559132b86230 2024-04-24T23:45:53.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1aced0 con 0x559132b8bd90 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51474/0 (socket says 172.21.15.107:51474) 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 -- 172.21.15.107:0/2497995723 learned_addr learned my addr 172.21.15.107:0/2497995723 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 0x559132b84580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a822b3640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b84ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 -- 172.21.15.107:0/2497995723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 msgr2=0x559132b84ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b84ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 -- 172.21.15.107:0/2497995723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 msgr2=0x559132b84580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 0x559132b84580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a812b1640 1 -- 172.21.15.107:0/2497995723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591328c87d0 con 0x559132b86c20 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.133+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 0x559132b84580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a812b1640 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0096c0 tx=0x7f0a6c0040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7503b647395b0771 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a677fe640 1 -- 172.21.15.107:0/2497995723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c035040 con 0x559132b86c20 2024-04-24T23:45:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a677fe640 1 -- 172.21.15.107:0/2497995723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6c047910 con 0x559132b86c20 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a677fe640 1 -- 172.21.15.107:0/2497995723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c004a00 con 0x559132b86c20 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2497995723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 msgr2=0x559132b978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0096c0 tx=0x7f0a6c0040f0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2497995723 shutdown_connections 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b84ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86c20 0x559132b978a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/2497995723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86230 0x559132b84580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.134+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2497995723 >> 172.21.15.107:0/2497995723 conn(0x559132b818a0 msgr2=0x559132b80b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2497995723 shutdown_connections 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- 172.21.15.107:0/2497995723 wait complete. 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- start start 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 Processor -- start 2024-04-24T23:45:53.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b95000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b96010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1adc70 con 0x559132b86230 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1ad5a0 con 0x559132b86c20 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1aa7e0 con 0x559132b8bd90 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a86a1a740 10 client.0 _ll_register_callbacks cb 0x7f0a780046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a81ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51490/0 (socket says 172.21.15.107:51490) 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3881730089 learned_addr learned my addr 172.21.15.107:0/3881730089 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a822b3640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b96010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.135+0000 7f0a812b1640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b95000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3881730089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 msgr2=0x559132b96010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b96010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 msgr2=0x559132b95000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b95000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 -- 172.21.15.107:0/3881730089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559132b96600 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a812b1640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b95000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a81ab2640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0a78097a80 tx=0x7f0a780cf490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7609e1b02d5a4fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7801d040 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0a78084bb0 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a6c047910 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a7809b620 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 10 client.15780 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559132b97970 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6c047910 con 0x559132b86230 2024-04-24T23:45:53.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0a7809a570 con 0x559132b86230 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 10 client.15780 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 10 client.15780 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 20 client.15780 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132c02b60 0x559132c04f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.136+0000 7f0a86a1a740 4 client.15780 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559132af1f00 con 0x559132c02b60 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a812b1640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132c02b60 0x559132c04f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a812b1640 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132c02b60 0x559132c04f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55efeec19e89672e server_cookie=527712d0c50dbe9a in_seq=0 out_seq=0 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a657fa640 10 client.15780 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a657fa640 1 -- 172.21.15.107:0/3881730089 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559132af1f00 con 0x559132c02b60 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a657fa640 10 client.15780 handle_command_reply: tid=0 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a86a1a740 1 client.15780 shutdown 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 msgr2=0x559132b94ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.137+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0a78097a80 tx=0x7f0a780cf490 comp rx=0 tx=0).stop 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 shutdown_connections 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559132c02b60 0x559132c04f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559132b8bd90 0x559132b96010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559132b86c20 0x559132b95000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 --2- 172.21.15.107:0/3881730089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559132b86230 0x559132b94ac0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a657fa640 0 client.15780 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 >> 172.21.15.107:0/3881730089 conn(0x559132b818a0 msgr2=0x559132b87620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 shutdown_connections 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 1 -- 172.21.15.107:0/3881730089 wait complete. 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a86a1a740 20 client.15780 trim_cache size 0 max 16384 2024-04-24T23:45:53.144 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:45:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 msgr2=0x7f0a680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 0x7f0a680a50d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0a74020000 tx=0x7f0a74021000 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 msgr2=0x7f0a7c104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c970 tx=0x7f0a7800ce30 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 shutdown_connections 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0a680a2c10 0x7f0a680a50d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0a7c102230 0x7f0a7c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a7c101870 0x7f0a7c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 --2- 172.21.15.107:0/3661811627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a7c100680 0x7f0a7c104610 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 >> 172.21.15.107:0/3661811627 conn(0x7f0a7c0fbbf0 msgr2=0x7f0a7c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 shutdown_connections 2024-04-24T23:45:53.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.138+0000 7f0a83d16640 1 -- 172.21.15.107:0/3661811627 wait complete. 2024-04-24T23:45:53.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 -- 172.21.15.107:0/1890581810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168107460 msgr2=0x7ff1681098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 --2- 172.21.15.107:0/1890581810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168107460 0x7ff1681098b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff160009ca0 tx=0x7ff16002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 -- 172.21.15.107:0/1890581810 shutdown_connections 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 --2- 172.21.15.107:0/1890581810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168107460 0x7ff1681098b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 --2- 172.21.15.107:0/1890581810 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168104b30 0x7ff168106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 --2- 172.21.15.107:0/1890581810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff1681045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 -- 172.21.15.107:0/1890581810 >> 172.21.15.107:0/1890581810 conn(0x7ff1680fbbf0 msgr2=0x7ff1680fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 -- 172.21.15.107:0/1890581810 shutdown_connections 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.253+0000 7ff16d238640 1 -- 172.21.15.107:0/1890581810 wait complete. 2024-04-24T23:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 Processor -- start 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 -- start start 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff168197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168107460 0x7ff16819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff16810d420 con 0x7ff168104b30 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.254+0000 7ff16d238640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff16810d2a0 con 0x7ff168102200 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16d238640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff16810d5a0 con 0x7ff168107460 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51502/0 (socket says 172.21.15.107:51502) 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 -- 172.21.15.107:0/745261138 learned_addr learned my addr 172.21.15.107:0/745261138 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff166d9d640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff168197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16759e640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168107460 0x7ff16819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 -- 172.21.15.107:0/745261138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168107460 msgr2=0x7ff16819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168107460 0x7ff16819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 -- 172.21.15.107:0/745261138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 msgr2=0x7ff168197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff168197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 -- 172.21.15.107:0/745261138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16819f680 con 0x7ff168104b30 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff166d9d640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff168197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16659c640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c970 tx=0x7ff15c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff14ffff640 1 -- 172.21.15.107:0/745261138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c007bb0 con 0x7ff168104b30 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16819f970 con 0x7ff168104b30 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.255+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff16819fec0 con 0x7ff168104b30 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.256+0000 7ff14ffff640 1 -- 172.21.15.107:0/745261138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff15c007d50 con 0x7ff168104b30 2024-04-24T23:45:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.256+0000 7ff14ffff640 1 -- 172.21.15.107:0/745261138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c018750 con 0x7ff168104b30 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 Processor -- start 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 -- start start 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b392045650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b39204d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b391fa9670 con 0x55b39204ad90 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b391f51970 con 0x55b392045230 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff170022740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b391fa5630 con 0x55b392045c20 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff14ffff640 1 -- 172.21.15.107:0/745261138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff15c018910 con 0x7ff168104b30 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff166d9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff16759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b392045650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.258+0000 7ff166d9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:33218/0 (socket says 172.21.15.107:33218) 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 -- 172.21.15.107:0/1726560596 learned_addr learned my addr 172.21.15.107:0/1726560596 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff16659c640 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b39204d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 -- 172.21.15.107:0/1726560596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 msgr2=0x55b392045650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b392045650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 -- 172.21.15.107:0/1726560596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 msgr2=0x55b39204d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b39204d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 -- 172.21.15.107:0/1726560596 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b391f75400 con 0x55b392045c20 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff16759e640 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b392045650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff166d9d640 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff158008c60 tx=0x7ff15800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd80e680fd3b1ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff14ffff640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 0x7ff1500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff16759e640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 0x7ff1500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1726560596 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158014070 con 0x55b392045c20 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1726560596 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1580047a0 con 0x55b392045c20 2024-04-24T23:45:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff14d7fa640 1 -- 172.21.15.107:0/1726560596 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158010740 con 0x55b392045c20 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.259+0000 7ff14ffff640 1 -- 172.21.15.107:0/745261138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff15c0cf3b0 con 0x7ff168104b30 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 -- 172.21.15.107:0/1726560596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 msgr2=0x55b39204a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff16759e640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 0x7ff1500a5050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55b392046480 tx=0x7ff160031bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff158008c60 tx=0x7ff15800e720 comp rx=0 tx=0).stop 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 -- 172.21.15.107:0/1726560596 shutdown_connections 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b39204d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045c20 0x55b39204a720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 --2- 172.21.15.107:0/1726560596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b392045650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 -- 172.21.15.107:0/1726560596 >> 172.21.15.107:0/1726560596 conn(0x55b3920408a0 msgr2=0x55b392040ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 -- 172.21.15.107:0/1726560596 shutdown_connections 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.260+0000 7ff170022740 1 -- 172.21.15.107:0/1726560596 wait complete. 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 -- start start 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 Processor -- start 2024-04-24T23:45:53.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045230 0x55b39204f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b39204f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b39204dd40 con 0x55b39204ad90 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b391f65050 con 0x55b392045c20 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b39204dec0 con 0x55b392045230 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff170022740 10 client.0 _ll_register_callbacks cb 0x55b391cbcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51516/0 (socket says 172.21.15.107:51516) 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 -- 172.21.15.107:0/2401251906 learned_addr learned my addr 172.21.15.107:0/2401251906 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16659c640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b39204f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff166d9d640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045230 0x55b39204f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 -- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045230 msgr2=0x55b39204f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045230 0x55b39204f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 -- 172.21.15.107:0/2401251906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 msgr2=0x55b39204f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b39204f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 -- 172.21.15.107:0/2401251906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b391f36dc0 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff16759e640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff160035000 tx=0x7ff16002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8141262d33606022 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.261+0000 7ff14aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1600045c0 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff160004760 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1580047a0 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff16002f610 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b391cbd3f0 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1580047a0 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff160030890 con 0x55b39204ad90 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 10 client.15792 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:45:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 20 client.15792 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b39205b2e0 0x55b39205d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 4 client.15792 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.262+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b391fb6bb0 con 0x55b39205b2e0 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.263+0000 7ff166d9d640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b39205b2e0 0x55b39205d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.263+0000 7ff14aff5640 10 client.15792 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.263+0000 7ff166d9d640 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b39205b2e0 0x55b39205d720 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=148b3f9431c9b527 server_cookie=41db699450748aee in_seq=0 out_seq=0 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff14aff5640 1 -- 172.21.15.107:0/2401251906 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b391fb6bb0 con 0x55b39205b2e0 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff14aff5640 10 client.15792 handle_command_reply: tid=0 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 client.15792 shutdown 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 msgr2=0x55b392056910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff160035000 tx=0x7ff16002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 shutdown_connections 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b39204ad90 0x55b392056910 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b39205b2e0 0x55b39205d720 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b39204f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 --2- 172.21.15.107:0/2401251906 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b392045230 0x55b39204f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 >> 172.21.15.107:0/2401251906 conn(0x55b3920408a0 msgr2=0x55b39203fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff14aff5640 0 client.15792 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 shutdown_connections 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 1 -- 172.21.15.107:0/2401251906 wait complete. 2024-04-24T23:45:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.264+0000 7ff170022740 20 client.15792 trim_cache size 0 max 16384 2024-04-24T23:45:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 Processor -- start 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 -- start start 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b39204ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b3920466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1540140e0 con 0x55b392045c20 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff154015c20 con 0x55b392045230 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff170022740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff1540147b0 con 0x55b39204ad90 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff16659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff16659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51532/0 (socket says 172.21.15.107:51532) 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.290+0000 7ff16659c640 1 -- 172.21.15.107:0/452989055 learned_addr learned my addr 172.21.15.107:0/452989055 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16759e640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b3920466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff166d9d640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b39204ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 -- 172.21.15.107:0/452989055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 msgr2=0x55b3920466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b3920466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 -- 172.21.15.107:0/452989055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 msgr2=0x55b39204ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b39204ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 -- 172.21.15.107:0/452989055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b391d877d0 con 0x55b392045c20 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff166d9d640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b39204ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff16659c640 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff15c004770 tx=0x7ff15c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee96c44c1e0b68d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff14cff9640 1 -- 172.21.15.107:0/452989055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c09bfe0 con 0x55b392045c20 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff14cff9640 1 -- 172.21.15.107:0/452989055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff15c00fc30 con 0x55b392045c20 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.291+0000 7ff14cff9640 1 -- 172.21.15.107:0/452989055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c099be0 con 0x55b392045c20 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- 172.21.15.107:0/452989055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 msgr2=0x55b392056830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff15c004770 tx=0x7ff15c00af90 comp rx=0 tx=0).stop 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- 172.21.15.107:0/452989055 shutdown_connections 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b3920466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045c20 0x55b392056830 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- 172.21.15.107:0/452989055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045230 0x55b39204ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- 172.21.15.107:0/452989055 >> 172.21.15.107:0/452989055 conn(0x55b3920408a0 msgr2=0x55b39204e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- 172.21.15.107:0/452989055 shutdown_connections 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- 172.21.15.107:0/452989055 wait complete. 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- start start 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 Processor -- start 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b392053f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b392054ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff154015550 con 0x55b392045230 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff154014e80 con 0x55b392045c20 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff1540120c0 con 0x55b39204ad90 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff170022740 10 client.0 _ll_register_callbacks cb 0x7ff160009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.292+0000 7ff166d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51534/0 (socket says 172.21.15.107:51534) 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 -- 172.21.15.107:0/211512768 learned_addr learned my addr 172.21.15.107:0/211512768 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff16759e640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b392054ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff16659c640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b392053f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 -- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 msgr2=0x55b392054ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b392054ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 -- 172.21.15.107:0/211512768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 msgr2=0x55b392053f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b392053f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 -- 172.21.15.107:0/211512768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3920554c0 con 0x55b392045230 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff16659c640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b392053f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff166d9d640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55b392046690 tx=0x7ff158008550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eec1df36394f651 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff14aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158014040 con 0x55b392045230 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff158003a00 con 0x55b392045230 2024-04-24T23:45:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff15c00fc30 con 0x55b392045230 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff158036050 con 0x55b392045230 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff170022740 10 client.15798 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3920567a0 con 0x55b392045230 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.293+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff15c00fc30 con 0x55b392045230 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff158004040 con 0x55b392045230 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 10 client.15798 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 10 client.15798 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 20 client.15798 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b3920c1a40 0x55b3920c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 4 client.15798 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b391fb0f00 con 0x55b3920c1a40 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff16659c640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b3920c1a40 0x55b3920c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff14aff5640 10 client.15798 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff16659c640 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b3920c1a40 0x55b3920c3e30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=643d1c0cbf06302d server_cookie=a9ea57f108c07a25 in_seq=0 out_seq=0 2024-04-24T23:45:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff14aff5640 1 -- 172.21.15.107:0/211512768 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b391fb0f00 con 0x55b3920c1a40 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff14aff5640 10 client.15798 handle_command_reply: tid=0 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.294+0000 7ff170022740 1 client.15798 shutdown 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 msgr2=0x55b3920539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55b392046690 tx=0x7ff158008550 comp rx=0 tx=0).stop 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 shutdown_connections 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b39204ad90 0x55b392054ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b3920c1a40 0x55b3920c3e30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b392045c20 0x55b392053f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 --2- 172.21.15.107:0/211512768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b392045230 0x55b3920539d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff14aff5640 0 client.15798 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 >> 172.21.15.107:0/211512768 conn(0x55b3920408a0 msgr2=0x55b39203f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 shutdown_connections 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 1 -- 172.21.15.107:0/211512768 wait complete. 2024-04-24T23:45:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff170022740 20 client.15798 trim_cache size 0 max 16384 2024-04-24T23:45:53.302 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:45:53.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 msgr2=0x7ff1500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 0x7ff1500a5050 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55b392046480 tx=0x7ff160031bb0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 msgr2=0x7ff168197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c970 tx=0x7ff15c00ce30 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 shutdown_connections 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff1500a2b90 0x7ff1500a5050 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff168107460 0x7ff16819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff168104b30 0x7ff168197fa0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 --2- 172.21.15.107:0/745261138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff168102200 0x7ff168197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.295+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 >> 172.21.15.107:0/745261138 conn(0x7ff1680fbbf0 msgr2=0x7ff168102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.296+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 shutdown_connections 2024-04-24T23:45:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.296+0000 7ff16d238640 1 -- 172.21.15.107:0/745261138 wait complete. 2024-04-24T23:45:53.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:53 smithi107 ceph-mon[29118]: pgmap v385: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:53.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:53 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3951264616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:53.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T23:45:53.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 -- 172.21.15.107:0/133179471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944104bf0 msgr2=0x7fb944106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/133179471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944104bf0 0x7fb944106fe0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb934009900 tx=0x7fb93402a410 comp rx=0 tx=0).stop 2024-04-24T23:45:53.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 -- 172.21.15.107:0/133179471 shutdown_connections 2024-04-24T23:45:53.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/133179471 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944107520 0x7fb944109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/133179471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944104bf0 0x7fb944106fe0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/133179471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 0x7fb9441046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 -- 172.21.15.107:0/133179471 >> 172.21.15.107:0/133179471 conn(0x7fb9440fbbf0 msgr2=0x7fb9440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 -- 172.21.15.107:0/133179471 shutdown_connections 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.413+0000 7fb94c3e7640 1 -- 172.21.15.107:0/133179471 wait complete. 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 Processor -- start 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 -- start start 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 0x7fb944197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944104bf0 0x7fb944198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb94410d410 con 0x7fb944107520 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb94410d290 con 0x7fb9441022c0 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94c3e7640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb94410d590 con 0x7fb944104bf0 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51552/0 (socket says 172.21.15.107:51552) 2024-04-24T23:45:53.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 -- 172.21.15.107:0/1108626788 learned_addr learned my addr 172.21.15.107:0/1108626788 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a183640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 0x7fb944197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb949982640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944104bf0 0x7fb944198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 -- 172.21.15.107:0/1108626788 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944104bf0 msgr2=0x7fb944198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944104bf0 0x7fb944198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 -- 172.21.15.107:0/1108626788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 msgr2=0x7fb944197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 0x7fb944197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.414+0000 7fb94a984640 1 -- 172.21.15.107:0/1108626788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94419f730 con 0x7fb944107520 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb94a984640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00eaa0 tx=0x7fb93c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb93b7fe640 1 -- 172.21.15.107:0/1108626788 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c00cdb0 con 0x7fb944107520 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb93b7fe640 1 -- 172.21.15.107:0/1108626788 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93c005020 con 0x7fb944107520 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94419fa20 con 0x7fb944107520 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb93b7fe640 1 -- 172.21.15.107:0/1108626788 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0116b0 con 0x7fb944107520 2024-04-24T23:45:53.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.415+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94419ff40 con 0x7fb944107520 2024-04-24T23:45:53.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.417+0000 7fb93b7fe640 1 -- 172.21.15.107:0/1108626788 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb93c005560 con 0x7fb944107520 2024-04-24T23:45:53.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.417+0000 7fb94f1d2740 1 Processor -- start 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 -- start start 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384dd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e38435670 con 0x562e384db1a0 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562e383dd970 con 0x562e384d6030 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562e38431630 con 0x562e384d5640 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb93b7fe640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 0x7fb9300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:33280/0 (socket says 172.21.15.107:33280) 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 -- 172.21.15.107:0/1159420554 learned_addr learned my addr 172.21.15.107:0/1159420554 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94a183640 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb94a984640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 0x7fb9300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb93b7fe640 1 -- 172.21.15.107:0/1108626788 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb93c0cf5d0 con 0x7fb944107520 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 -- 172.21.15.107:0/1159420554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 msgr2=0x562e384dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 -- 172.21.15.107:0/1159420554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 msgr2=0x562e384dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384dd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.418+0000 7fb949982640 1 -- 172.21.15.107:0/1159420554 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e38401400 con 0x562e384d5640 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.419+0000 7fb949982640 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9340096c0 tx=0x7fb934005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=925a2d08f31f0144 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.419+0000 7fb94a984640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 0x7fb9300a5050 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0984e0 tx=0x7fb93c01eb40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.419+0000 7fb938bfe640 1 -- 172.21.15.107:0/1159420554 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934035070 con 0x562e384d5640 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.419+0000 7fb938bfe640 1 -- 172.21.15.107:0/1159420554 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb934005510 con 0x562e384d5640 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.419+0000 7fb938bfe640 1 -- 172.21.15.107:0/1159420554 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934004ae0 con 0x562e384d5640 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- 172.21.15.107:0/1159420554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 msgr2=0x562e384d5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9340096c0 tx=0x7fb934005fd0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- 172.21.15.107:0/1159420554 shutdown_connections 2024-04-24T23:45:53.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384dd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/1159420554 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384d5a60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- 172.21.15.107:0/1159420554 >> 172.21.15.107:0/1159420554 conn(0x562e384cc8a0 msgr2=0x562e384ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- 172.21.15.107:0/1159420554 shutdown_connections 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- 172.21.15.107:0/1159420554 wait complete. 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- start start 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 Processor -- start 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384db1a0 0x562e384e6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e384de0f0 con 0x562e384d5640 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562e383f1050 con 0x562e384d6030 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562e384de270 con 0x562e384db1a0 2024-04-24T23:45:53.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb94f1d2740 10 client.0 _ll_register_callbacks cb 0x562e38148be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb949982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb949982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51570/0 (socket says 172.21.15.107:51570) 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb949982640 1 -- 172.21.15.107:0/4146005142 learned_addr learned my addr 172.21.15.107:0/4146005142 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.420+0000 7fb949982640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384db1a0 0x562e384e6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb94a183640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 -- 172.21.15.107:0/4146005142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384db1a0 msgr2=0x562e384e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384db1a0 0x562e384e6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 -- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 msgr2=0x562e384dfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 -- 172.21.15.107:0/4146005142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e383c2dc0 con 0x562e384d5640 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb949982640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb934031610 tx=0x7fb934032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aae5a808eb7a051 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb92dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934035050 con 0x562e384d5640 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb934003a00 con 0x562e384d5640 2024-04-24T23:45:53.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb934005510 con 0x562e384d5640 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb93405a050 con 0x562e384d5640 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb94f1d2740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.421+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e381493f0 con 0x562e384d5640 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb934005510 con 0x562e384d5640 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb93404da10 con 0x562e384d5640 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 20 client.15810 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e384eb200 0x562e384ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 4 client.15810 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e38442bb0 con 0x562e384eb200 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.422+0000 7fb94a183640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e384eb200 0x562e384ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb92dffb640 10 client.15810 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94a183640 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e384eb200 0x562e384ed5f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32cb14c51a558758 server_cookie=5b394320abe8aeb6 in_seq=0 out_seq=0 2024-04-24T23:45:53.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb92dffb640 1 -- 172.21.15.107:0/4146005142 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e38442bb0 con 0x562e384eb200 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb92dffb640 10 client.15810 handle_command_reply: tid=0 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 client.15810 shutdown 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 msgr2=0x562e384df790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb934031610 tx=0x7fb934032f90 comp rx=0 tx=0).stop 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 shutdown_connections 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e384eb200 0x562e384ed5f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384db1a0 0x562e384e6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb92dffb640 0 client.15810 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384dfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4146005142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d5640 0x562e384df790 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.423+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 >> 172.21.15.107:0/4146005142 conn(0x562e384cc8a0 msgr2=0x562e384cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.424+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 shutdown_connections 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.424+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4146005142 wait complete. 2024-04-24T23:45:53.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.424+0000 7fb94f1d2740 20 client.15810 trim_cache size 0 max 16384 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 Processor -- start 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 -- start start 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a82b0 con 0x562e384db1a0 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a9df0 con 0x562e384d6030 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.437+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a8980 con 0x562e384d5640 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51580/0 (socket says 172.21.15.107:51580) 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 -- 172.21.15.107:0/3933150548 learned_addr learned my addr 172.21.15.107:0/3933150548 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb94a183640 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 -- 172.21.15.107:0/3933150548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 msgr2=0x562e384cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 -- 172.21.15.107:0/3933150548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 msgr2=0x562e384e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384e6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 -- 172.21.15.107:0/3933150548 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e382137d0 con 0x562e384db1a0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb949982640 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x562e384e8170 tx=0x7fb934005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb81ea1bae932b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb92ffff640 1 -- 172.21.15.107:0/3933150548 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934063070 con 0x562e384db1a0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb92ffff640 1 -- 172.21.15.107:0/3933150548 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93404be00 con 0x562e384db1a0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.438+0000 7fb92ffff640 1 -- 172.21.15.107:0/3933150548 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934035040 con 0x562e384db1a0 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- 172.21.15.107:0/3933150548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 msgr2=0x562e384cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x562e384e8170 tx=0x7fb934005990 comp rx=0 tx=0).stop 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- 172.21.15.107:0/3933150548 shutdown_connections 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384db1a0 0x562e384cb7f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384d6030 0x562e384e6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/3933150548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- 172.21.15.107:0/3933150548 >> 172.21.15.107:0/3933150548 conn(0x562e384cc8a0 msgr2=0x562e384d7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- 172.21.15.107:0/3933150548 shutdown_connections 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- 172.21.15.107:0/3933150548 wait complete. 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- start start 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 Processor -- start 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384db1a0 0x562e384e5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a9720 con 0x562e384d6030 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a9050 con 0x562e384db1a0 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441a6290 con 0x562e384d5640 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.439+0000 7fb94f1d2740 10 client.0 _ll_register_callbacks cb 0x7fb934032e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51586/0 (socket says 172.21.15.107:51586) 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 -- 172.21.15.107:0/4286050164 learned_addr learned my addr 172.21.15.107:0/4286050164 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb949982640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb949982640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384db1a0 0x562e384e5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 -- 172.21.15.107:0/4286050164 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 msgr2=0x562e384e45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384e45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 -- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384db1a0 msgr2=0x562e384e5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384db1a0 0x562e384e5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 -- 172.21.15.107:0/4286050164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e384e60f0 con 0x562e384d6030 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94a183640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb940004700 tx=0x7fb94001da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3269bc10adcd8547 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb92dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940011d60 con 0x562e384d6030 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb940004020 con 0x562e384d6030 2024-04-24T23:45:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb93404be00 con 0x562e384d6030 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb940004220 con 0x562e384d6030 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94f1d2740 10 client.15816 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.440+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e385518e0 con 0x562e384d6030 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93404be00 con 0x562e384d6030 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb940025540 con 0x562e384d6030 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 10 client.15816 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 20 client.15816 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e385522c0 0x562e385546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 4 client.15816 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e3843cf00 con 0x562e385522c0 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.441+0000 7fb949982640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e385522c0 0x562e385546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb949982640 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e385522c0 0x562e385546e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3522a0781420847d server_cookie=2187b4b6a2b445f2 in_seq=0 out_seq=0 2024-04-24T23:45:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb92dffb640 10 client.15816 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb92dffb640 1 -- 172.21.15.107:0/4286050164 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e3843cf00 con 0x562e385522c0 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb92dffb640 10 client.15816 handle_command_reply: tid=0 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 client.15816 shutdown 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 msgr2=0x562e384e4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb940004700 tx=0x7fb94001da40 comp rx=0 tx=0).stop 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 shutdown_connections 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562e385522c0 0x562e385546e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562e384db1a0 0x562e384e5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e384d6030 0x562e384e4af0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb92dffb640 0 client.15816 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 --2- 172.21.15.107:0/4286050164 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562e384d5640 0x562e384e45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 >> 172.21.15.107:0/4286050164 conn(0x562e384cc8a0 msgr2=0x562e384cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 shutdown_connections 2024-04-24T23:45:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 1 -- 172.21.15.107:0/4286050164 wait complete. 2024-04-24T23:45:53.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.442+0000 7fb94f1d2740 20 client.15816 trim_cache size 0 max 16384 2024-04-24T23:45:53.449 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 msgr2=0x7fb9300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 0x7fb9300a5050 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0984e0 tx=0x7fb93c01eb40 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 msgr2=0x7fb94419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00eaa0 tx=0x7fb93c00ef60 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 shutdown_connections 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb9300a2b90 0x7fb9300a5050 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb944107520 0x7fb94419f0c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb944104bf0 0x7fb944198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 --2- 172.21.15.107:0/1108626788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9441022c0 0x7fb944197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 >> 172.21.15.107:0/1108626788 conn(0x7fb9440fbbf0 msgr2=0x7fb944102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 shutdown_connections 2024-04-24T23:45:53.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:53.443+0000 7fb94c3e7640 1 -- 172.21.15.107:0/1108626788 wait complete. 2024-04-24T23:45:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:53 smithi195 ceph-mon[33019]: pgmap v385: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:53.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:53 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3951264616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:53 smithi170 ceph-mon[32714]: pgmap v385: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:45:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:53 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3951264616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:45:54.454 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T23:45:54.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:45:54.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T23:45:54.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.560+0000 7fdfac39e640 1 -- 172.21.15.107:0/3782153672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4100680 msgr2=0x7fdfa4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.560+0000 7fdfac39e640 1 --2- 172.21.15.107:0/3782153672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4100680 0x7fdfa4100aa0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009900 tx=0x7fdfa002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- 172.21.15.107:0/3782153672 shutdown_connections 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- 172.21.15.107:0/3782153672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4102230 0x7fdfa4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- 172.21.15.107:0/3782153672 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- 172.21.15.107:0/3782153672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4100680 0x7fdfa4100aa0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- 172.21.15.107:0/3782153672 >> 172.21.15.107:0/3782153672 conn(0x7fdfa40fbbf0 msgr2=0x7fdfa40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- 172.21.15.107:0/3782153672 shutdown_connections 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- 172.21.15.107:0/3782153672 wait complete. 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 Processor -- start 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- start start 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4100680 0x7fdfa4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410c2d0 con 0x7fdfa4102230 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410c150 con 0x7fdfa4100680 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfac39e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410c450 con 0x7fdfa4101870 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfaa93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfaa93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51602/0 (socket says 172.21.15.107:51602) 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.561+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1665278652 learned_addr learned my addr 172.21.15.107:0/1665278652 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4100680 0x7fdfa4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1665278652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 msgr2=0x7fdfa4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1665278652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4100680 msgr2=0x7fdfa4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4100680 0x7fdfa4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1665278652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa419f670 con 0x7fdfa4102230 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:54.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00cce0 tx=0x7fdf9c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdf9b7fe640 1 -- 172.21.15.107:0/1665278652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c00f450 con 0x7fdfa4102230 2024-04-24T23:45:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdf9b7fe640 1 -- 172.21.15.107:0/1665278652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c00fad0 con 0x7fdfa4102230 2024-04-24T23:45:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdf9b7fe640 1 -- 172.21.15.107:0/1665278652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c018c10 con 0x7fdfa4102230 2024-04-24T23:45:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa419f960 con 0x7fdfa4102230 2024-04-24T23:45:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.562+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa419fe60 con 0x7fdfa4102230 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.565+0000 7fdf9b7fe640 1 -- 172.21.15.107:0/1665278652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdf9c00f5f0 con 0x7fdfa4102230 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.565+0000 7fdfaf186740 1 Processor -- start 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 -- start start 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdf9b7fe640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 0x7fdf900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd996065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd99606b90 0x55fd9960b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fd995652c0 con 0x55fd9960bd00 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fd9950d910 con 0x55fd996061a0 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaf186740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fd99561280 con 0x55fd99606b90 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdf9b7fe640 1 -- 172.21.15.107:0/1665278652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdf9c0cfba0 con 0x7fdfa4102230 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 0x7fdf900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51604/0 (socket says 172.21.15.107:51604) 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1674240058 learned_addr learned my addr 172.21.15.107:0/1674240058 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd996065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1674240058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd99606b90 msgr2=0x55fd9960b690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd99606b90 0x55fd9960b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1674240058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 msgr2=0x55fd996065c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd996065c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.566+0000 7fdfaa93b640 1 -- 172.21.15.107:0/1674240058 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd9951efb0 con 0x55fd9960bd00 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 0x7fdf900a5050 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4198f60 tx=0x7fdf94005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c09bd30 tx=0x7fdf9c09bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43490fafd626a2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdf98bfe640 1 -- 172.21.15.107:0/1674240058 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c04a790 con 0x55fd9960bd00 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdf98bfe640 1 -- 172.21.15.107:0/1674240058 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c09c730 con 0x55fd9960bd00 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdf98bfe640 1 -- 172.21.15.107:0/1674240058 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c09a330 con 0x55fd9960bd00 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdfaf186740 1 -- 172.21.15.107:0/1674240058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 msgr2=0x55fd9960e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.567+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c09bd30 tx=0x7fdf9c09bc30 comp rx=0 tx=0).stop 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- 172.21.15.107:0/1674240058 shutdown_connections 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd9960bd00 0x55fd9960e190 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd99606b90 0x55fd9960b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1674240058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd996065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- 172.21.15.107:0/1674240058 >> 172.21.15.107:0/1674240058 conn(0x55fd995c9950 msgr2=0x55fd995fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- 172.21.15.107:0/1674240058 shutdown_connections 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- 172.21.15.107:0/1674240058 wait complete. 2024-04-24T23:45:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- start start 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 Processor -- start 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd99606b90 0x55fd99610b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fd9960ecb0 con 0x55fd996061a0 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fd99520e90 con 0x55fd99606b90 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fd9960ee30 con 0x55fd9960bd00 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaf186740 10 client.0 _ll_register_callbacks cb 0x55fd9955c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaa13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaa13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51618/0 (socket says 172.21.15.107:51618) 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaa13a640 1 -- 172.21.15.107:0/3605398502 learned_addr learned my addr 172.21.15.107:0/3605398502 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.568+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd99606b90 0x55fd99610b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 -- 172.21.15.107:0/3605398502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 msgr2=0x55fd99617ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 -- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd99606b90 msgr2=0x55fd99610b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd99606b90 0x55fd99610b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 -- 172.21.15.107:0/3605398502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd993417e0 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0031440 tx=0x7fdfa0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf9bc8861adcb16 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0035070 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdfa00035f0 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf9c09c730 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdfa00037f0 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaf186740 10 client.15828 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fd993a1f70 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c09c730 con 0x55fd996061a0 2024-04-24T23:45:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.569+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdfa0044cd0 con 0x55fd996061a0 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 10 client.15828 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 20 client.15828 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd9961c140 0x55fd9961e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 4 client.15828 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd995395a0 con 0x55fd9961c140 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd9961c140 0x55fd9961e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd9961c140 0x55fd9961e5c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=333b106908793133 server_cookie=b1f5de5b41dddcaf in_seq=0 out_seq=0 2024-04-24T23:45:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.570+0000 7fdf8dffb640 10 client.15828 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdf8dffb640 1 -- 172.21.15.107:0/3605398502 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd995395a0 con 0x55fd9961c140 2024-04-24T23:45:54.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdf8dffb640 10 client.15828 handle_command_reply: tid=0 2024-04-24T23:45:54.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 client.15828 shutdown 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 msgr2=0x55fd99610620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0031440 tx=0x7fdfa0004270 comp rx=0 tx=0).stop 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 shutdown_connections 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd9961c140 0x55fd9961e5c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdf8dffb640 0 client.15828 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd99606b90 0x55fd99610b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3605398502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd996061a0 0x55fd99610620 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 >> 172.21.15.107:0/3605398502 conn(0x55fd995c9950 msgr2=0x55fd995fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 shutdown_connections 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 1 -- 172.21.15.107:0/3605398502 wait complete. 2024-04-24T23:45:54.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.571+0000 7fdfaf186740 20 client.15828 trim_cache size 0 max 16384 2024-04-24T23:45:54.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 Processor -- start 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 -- start start 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd996061a0 0x55fd995fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 0x55fd99617ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a8170 con 0x55fd99606b90 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a9cb0 con 0x55fd9960bd00 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaf186740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a8840 con 0x55fd996061a0 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51624/0 (socket says 172.21.15.107:51624) 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 -- 172.21.15.107:0/3699639442 learned_addr learned my addr 172.21.15.107:0/3699639442 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd996061a0 0x55fd995fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 0x55fd99617ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 -- 172.21.15.107:0/3699639442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd996061a0 msgr2=0x55fd995fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd996061a0 0x55fd995fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 -- 172.21.15.107:0/3699639442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 msgr2=0x55fd99617ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 0x55fd99617ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 -- 172.21.15.107:0/3699639442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd99239170 con 0x55fd99606b90 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 0x55fd99617ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.589+0000 7fdfa9939640 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400d710 tx=0x7fdf94013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c75ddbd0aa142e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdf8ffff640 1 -- 172.21.15.107:0/3699639442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94024b50 con 0x55fd99606b90 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdf8ffff640 1 -- 172.21.15.107:0/3699639442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94024cf0 con 0x55fd99606b90 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdf8ffff640 1 -- 172.21.15.107:0/3699639442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9400fa00 con 0x55fd99606b90 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 -- 172.21.15.107:0/3699639442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 msgr2=0x55fd995fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400d710 tx=0x7fdf94013000 comp rx=0 tx=0).stop 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 -- 172.21.15.107:0/3699639442 shutdown_connections 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd9960bd00 0x55fd99617ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd995fcd10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 --2- 172.21.15.107:0/3699639442 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd996061a0 0x55fd995fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 -- 172.21.15.107:0/3699639442 >> 172.21.15.107:0/3699639442 conn(0x55fd995c9950 msgr2=0x55fd995fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 -- 172.21.15.107:0/3699639442 shutdown_connections 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.590+0000 7fdfaf186740 1 -- 172.21.15.107:0/3699639442 wait complete. 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 -- start start 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 Processor -- start 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd99615bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a95e0 con 0x55fd99606b90 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a8f10 con 0x55fd996061a0 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa41a6150 con 0x55fd9960bd00 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaf186740 10 client.0 _ll_register_callbacks cb 0x7fdfa0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51630/0 (socket says 172.21.15.107:51630) 2024-04-24T23:45:54.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 -- 172.21.15.107:0/1096503179 learned_addr learned my addr 172.21.15.107:0/1096503179 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaa93b640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 -- 172.21.15.107:0/1096503179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 msgr2=0x55fd99617120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd99615bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 -- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 msgr2=0x55fd99615bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd99615bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfa9939640 1 -- 172.21.15.107:0/1096503179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd99617710 con 0x55fd99606b90 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.591+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd99615bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfa9939640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400f500 tx=0x7fdf94024c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ac95e7a89c7ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94039060 con 0x55fd99606b90 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdf94005040 con 0x55fd99606b90 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf94024cf0 con 0x55fd99606b90 2024-04-24T23:45:54.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdf94040050 con 0x55fd99606b90 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 10 client.15834 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fd99682fd0 con 0x55fd99606b90 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94024cf0 con 0x55fd99606b90 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdf940046c0 con 0x55fd99606b90 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 10 client.15834 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 20 client.15834 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd99683940 0x55fd99685d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 4 client.15834 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.592+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fd99572800 con 0x55fd99683940 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.593+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd99683940 0x55fd99685d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.593+0000 7fdf8dffb640 10 client.15834 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.593+0000 7fdfaa13a640 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd99683940 0x55fd99685d60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d252f685c0253cf7 server_cookie=d78ded896f9773d7 in_seq=0 out_seq=0 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdf8dffb640 1 -- 172.21.15.107:0/1096503179 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fd99572800 con 0x55fd99683940 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdf8dffb640 10 client.15834 handle_command_reply: tid=0 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdfaf186740 1 client.15834 shutdown 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 msgr2=0x55fd99616110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400f500 tx=0x7fdf94024c40 comp rx=0 tx=0).stop 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.594+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 shutdown_connections 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55fd99683940 0x55fd99685d60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fd9960bd00 0x55fd99617120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fd99606b90 0x55fd99616110 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdf8dffb640 0 client.15834 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 --2- 172.21.15.107:0/1096503179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fd996061a0 0x55fd99615bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 >> 172.21.15.107:0/1096503179 conn(0x55fd995c9950 msgr2=0x55fd995fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 shutdown_connections 2024-04-24T23:45:54.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 1 -- 172.21.15.107:0/1096503179 wait complete. 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfaf186740 20 client.15834 trim_cache size 0 max 16384 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "6280a320-8b22-4562-93fe-aa1d525cebbf", 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-04-24T23:45:54.601 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 msgr2=0x7fdf900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 0x7fdf900a5050 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4198f60 tx=0x7fdf94005d90 comp rx=0 tx=0).stop 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 msgr2=0x7fdfa419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00cce0 tx=0x7fdf9c00a620 comp rx=0 tx=0).stop 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 shutdown_connections 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdf900a2b90 0x7fdf900a5050 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa4102230 0x7fdfa419f000 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdfa4101870 0x7fdfa4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 --2- 172.21.15.107:0/1665278652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa4100680 0x7fdfa4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 >> 172.21.15.107:0/1665278652 conn(0x7fdfa40fbbf0 msgr2=0x7fdfa40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 shutdown_connections 2024-04-24T23:45:54.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.595+0000 7fdfac39e640 1 -- 172.21.15.107:0/1665278652 wait complete. 2024-04-24T23:45:54.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:45:54.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- 172.21.15.107:0/783787179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4101790 msgr2=0x7f50f4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 --2- 172.21.15.107:0/783787179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4101790 0x7f50f4101bb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009900 tx=0x7f50f002a410 comp rx=0 tx=0).stop 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- 172.21.15.107:0/783787179 shutdown_connections 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 --2- 172.21.15.107:0/783787179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4103340 0x7f50f4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 --2- 172.21.15.107:0/783787179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4102980 0x7f50f4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 --2- 172.21.15.107:0/783787179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4101790 0x7f50f4101bb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- 172.21.15.107:0/783787179 >> 172.21.15.107:0/783787179 conn(0x7f50f40fcd20 msgr2=0x7f50f40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- 172.21.15.107:0/783787179 shutdown_connections 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- 172.21.15.107:0/783787179 wait complete. 2024-04-24T23:45:54.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 Processor -- start 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.709+0000 7f50fc509640 1 -- start start 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4101790 0x7f50f4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4103340 0x7f50f419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d450 con 0x7f50f4102980 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d2d0 con 0x7f50f4101790 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d5d0 con 0x7f50f4103340 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51640/0 (socket says 172.21.15.107:51640) 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 -- 172.21.15.107:0/2978558205 learned_addr learned my addr 172.21.15.107:0/2978558205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50faaa6640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4103340 0x7f50f419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4101790 0x7f50f4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 -- 172.21.15.107:0/2978558205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4103340 msgr2=0x7f50f419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4103340 0x7f50f419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 -- 172.21.15.107:0/2978558205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4101790 msgr2=0x7f50f4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4101790 0x7f50f4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 -- 172.21.15.107:0/2978558205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f419f6f0 con 0x7f50f4102980 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f50e400e9f0 tx=0x7f50e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50eb7fe640 1 -- 172.21.15.107:0/2978558205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e400cd80 con 0x7f50f4102980 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f419f9e0 con 0x7f50f4102980 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.710+0000 7f50eb7fe640 1 -- 172.21.15.107:0/2978558205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e4005500 con 0x7f50f4102980 2024-04-24T23:45:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.711+0000 7f50eb7fe640 1 -- 172.21.15.107:0/2978558205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4010690 con 0x7f50f4102980 2024-04-24T23:45:54.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.711+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f419fee0 con 0x7f50f4102980 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.713+0000 7f50eb7fe640 1 -- 172.21.15.107:0/2978558205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f50e40046c0 con 0x7f50f4102980 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 Processor -- start 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 -- start start 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50eb7fe640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 0x7f50e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407f1b0 0x55b004081640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b003fd96b0 con 0x55b004079650 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b003f81880 con 0x55b00407f1b0 2024-04-24T23:45:54.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50ff20f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b003fd5670 con 0x55b00407a040 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 0x7f50e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51642/0 (socket says 172.21.15.107:51642) 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50faaa6640 1 -- 172.21.15.107:0/2453061231 learned_addr learned my addr 172.21.15.107:0/2453061231 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50eb7fe640 1 -- 172.21.15.107:0/2978558205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f50e40cf810 con 0x7f50f4102980 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.714+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407f1b0 0x55b004081640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 -- 172.21.15.107:0/2453061231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 msgr2=0x55b00407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 -- 172.21.15.107:0/2453061231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407f1b0 msgr2=0x55b004081640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407f1b0 0x55b004081640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 -- 172.21.15.107:0/2453061231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b003db77d0 con 0x55b004079650 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 0x7f50e00a5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f50f002aa00 tx=0x7f50f00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50faaa6640 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f50ec014930 tx=0x7f50ec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808b6e1501a4e078 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50e8bfe640 1 -- 172.21.15.107:0/2453061231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec014050 con 0x55b004079650 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50e8bfe640 1 -- 172.21.15.107:0/2453061231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec0141f0 con 0x55b004079650 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.715+0000 7f50e8bfe640 1 -- 172.21.15.107:0/2453061231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec0052a0 con 0x55b004079650 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- 172.21.15.107:0/2453061231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 msgr2=0x55b004079a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f50ec014930 tx=0x7f50ec009f90 comp rx=0 tx=0).stop 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- 172.21.15.107:0/2453061231 shutdown_connections 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407f1b0 0x55b004081640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- 172.21.15.107:0/2453061231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004079a70 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- 172.21.15.107:0/2453061231 >> 172.21.15.107:0/2453061231 conn(0x55b0040708b0 msgr2=0x55b004070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- 172.21.15.107:0/2453061231 shutdown_connections 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- 172.21.15.107:0/2453061231 wait complete. 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- start start 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 Processor -- start 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b004083f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.716+0000 7f50ff20f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b004082160 con 0x55b00407f1b0 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b003f94f50 con 0x55b004079650 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b0040822e0 con 0x55b00407a040 2024-04-24T23:45:54.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 10 client.0 _ll_register_callbacks cb 0x55b003fd0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51648/0 (socket says 172.21.15.107:51648) 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 -- 172.21.15.107:0/601870731 learned_addr learned my addr 172.21.15.107:0/601870731 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b004083f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 -- 172.21.15.107:0/601870731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 msgr2=0x55b004083f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b004083f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 -- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 msgr2=0x55b004083a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 -- 172.21.15.107:0/601870731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b003ced3f0 con 0x55b00407f1b0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50faaa6640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f50ec01c040 tx=0x7f50ec012b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b00d6ad7c1a9d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec029640 con 0x55b00407f1b0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50ec0141f0 con 0x55b00407f1b0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f50ec01f430 con 0x55b00407f1b0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50ec0297e0 con 0x55b00407f1b0 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:54.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.717+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b003cad0f0 con 0x55b00407f1b0 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec0141f0 con 0x55b00407f1b0 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50ec027070 con 0x55b00407f1b0 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 20 client.15846 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b00408f680 0x55b004091ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 4 client.15846 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b003fe6bf0 con 0x55b00408f680 2024-04-24T23:45:54.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.718+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b00408f680 0x55b004091ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b00408f680 0x55b004091ab0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d60604e0f24c473 server_cookie=f6a75decf1753ca6 in_seq=0 out_seq=0 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ddffb640 10 client.15846 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ddffb640 1 -- 172.21.15.107:0/601870731 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b003fe6bf0 con 0x55b00408f680 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ddffb640 10 client.15846 handle_command_reply: tid=0 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 client.15846 shutdown 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 msgr2=0x55b00408afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f50ec01c040 tx=0x7f50ec012b40 comp rx=0 tx=0).stop 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 shutdown_connections 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b00408f680 0x55b004091ab0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b00408afd0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ddffb640 0 client.15846 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b004083f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 --2- 172.21.15.107:0/601870731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 >> 172.21.15.107:0/601870731 conn(0x55b0040708b0 msgr2=0x55b00406faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 shutdown_connections 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.719+0000 7f50ff20f740 1 -- 172.21.15.107:0/601870731 wait complete. 2024-04-24T23:45:54.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.720+0000 7f50ff20f740 20 client.15846 trim_cache size 0 max 16384 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 Processor -- start 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 -- start start 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00408af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a82d0 con 0x55b00407f1b0 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a9e10 con 0x55b004079650 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.737+0000 7f50ff20f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a89a0 con 0x55b00407a040 2024-04-24T23:45:54.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51662/0 (socket says 172.21.15.107:51662) 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 -- 172.21.15.107:0/4231136901 learned_addr learned my addr 172.21.15.107:0/4231136901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00408af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 -- 172.21.15.107:0/4231136901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 msgr2=0x55b00408af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00408af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 -- 172.21.15.107:0/4231136901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 msgr2=0x55b004083a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 -- 172.21.15.107:0/4231136901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b004071620 con 0x55b00407f1b0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50faaa6640 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009c30 tx=0x7f50ec009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2885eb6e3e6d0d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50dffff640 1 -- 172.21.15.107:0/4231136901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec014610 con 0x55b00407f1b0 2024-04-24T23:45:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50dffff640 1 -- 172.21.15.107:0/4231136901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec0283e0 con 0x55b00407f1b0 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.738+0000 7f50dffff640 1 -- 172.21.15.107:0/4231136901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec003970 con 0x55b00407f1b0 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- 172.21.15.107:0/4231136901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 msgr2=0x55b004083f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009c30 tx=0x7f50ec009e70 comp rx=0 tx=0).stop 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- 172.21.15.107:0/4231136901 shutdown_connections 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00407f1b0 0x55b004083f90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407a040 0x55b00408af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4231136901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b004079650 0x55b004083a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- 172.21.15.107:0/4231136901 >> 172.21.15.107:0/4231136901 conn(0x55b0040708b0 msgr2=0x55b00407b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- 172.21.15.107:0/4231136901 shutdown_connections 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- 172.21.15.107:0/4231136901 wait complete. 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 -- start start 2024-04-24T23:45:54.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 Processor -- start 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.739+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 0x55b004088170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407f1b0 0x55b0040f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a9740 con 0x55b004079650 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a9070 con 0x55b00407a040 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f50f41a62b0 con 0x55b00407f1b0 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 10 client.0 _ll_register_callbacks cb 0x7f50e400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51674/0 (socket says 172.21.15.107:51674) 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 -- 172.21.15.107:0/4289211138 learned_addr learned my addr 172.21.15.107:0/4289211138 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 0x55b004088170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 -- 172.21.15.107:0/4289211138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407f1b0 msgr2=0x55b0040f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50faaa6640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407f1b0 0x55b0040f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407f1b0 0x55b0040f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 -- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 msgr2=0x55b004088170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 0x55b004088170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 -- 172.21.15.107:0/4289211138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b004088760 con 0x55b004079650 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 0x55b004088170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50fa2a5640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f50f00044a0 tx=0x7f50f003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40471dcc78cf497f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0035040 con 0x55b004079650 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f50f0005200 con 0x55b004079650 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50ec0283e0 con 0x55b004079650 2024-04-24T23:45:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50f0061050 con 0x55b004079650 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 10 client.15852 fetch_fsmap learned FSMap version 29 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.740+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0040f5210 con 0x55b004079650 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec0283e0 con 0x55b004079650 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50f00327c0 con 0x55b004079650 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 10 client.15852 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 20 client.15852 populate_metadata read hostname 'smithi107' 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b0040f5be0 0x55b0040f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 4 client.15852 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b004083840 con 0x55b0040f5be0 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b0040f5be0 0x55b0040f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50f9aa4640 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b0040f5be0 0x55b0040f8000 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=defecc8b5855f80d server_cookie=d9b6875a5609df78 in_seq=0 out_seq=0 2024-04-24T23:45:54.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.741+0000 7f50ddffb640 10 client.15852 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ddffb640 1 -- 172.21.15.107:0/4289211138 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b004083840 con 0x55b0040f5be0 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ddffb640 10 client.15852 handle_command_reply: tid=0 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 client.15852 shutdown 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 msgr2=0x55b004087c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f50f00044a0 tx=0x7f50f003d000 comp rx=0 tx=0).stop 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 shutdown_connections 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55b0040f5be0 0x55b0040f8000 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b00407f1b0 0x55b0040f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b00407a040 0x55b004088170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 --2- 172.21.15.107:0/4289211138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b004079650 0x55b004087c30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ddffb640 0 client.15852 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:45:54.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 >> 172.21.15.107:0/4289211138 conn(0x55b0040708b0 msgr2=0x55b004083a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 shutdown_connections 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 1 -- 172.21.15.107:0/4289211138 wait complete. 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50ff20f740 20 client.15852 trim_cache size 0 max 16384 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout: "6280a320-8b22-4562-93fe-aa1d525cebbf": { 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-04-24T23:45:54.750 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "6280a320-8b22-4562-93fe-aa1d525cebbf", 2024-04-24T23:45:54.751 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-04-24T23:45:54.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:45:54.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:45:54.751 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:45:54.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 msgr2=0x7f50e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 0x7f50e00a5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f50f002aa00 tx=0x7f50f00315b0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 msgr2=0x7f50f4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f50e400e9f0 tx=0x7f50e400edf0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 shutdown_connections 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f50e00a2b90 0x7f50e00a5050 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50f4103340 0x7f50f419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f4102980 0x7f50f4198010 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 --2- 172.21.15.107:0/2978558205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50f4101790 0x7f50f4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.742+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 >> 172.21.15.107:0/2978558205 conn(0x7f50f40fcd20 msgr2=0x7f50f4105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:45:54.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.743+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 shutdown_connections 2024-04-24T23:45:54.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:45:54.743+0000 7f50fc509640 1 -- 172.21.15.107:0/2978558205 wait complete. 2024-04-24T23:45:54.754 INFO:tasks.cephfs.filesystem:scrub status for tag:6280a320-8b22-4562-93fe-aa1d525cebbf - {'path': '/', 'tag': '6280a320-8b22-4562-93fe-aa1d525cebbf', 'options': 'recursive,force'} 2024-04-24T23:45:55.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:55 smithi195 ceph-mon[33019]: pgmap v386: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:55 smithi170 ceph-mon[32714]: pgmap v386: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:55.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:55 smithi107 ceph-mon[29118]: pgmap v386: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:45:56.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:56 smithi195 ceph-mon[33019]: scrub queued for path: / 2024-04-24T23:45:56.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:56 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:56.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:56 smithi195 ceph-mon[33019]: scrub summary: active paths [/] 2024-04-24T23:45:56.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:56 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:56 smithi170 ceph-mon[32714]: scrub queued for path: / 2024-04-24T23:45:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:56 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:56 smithi170 ceph-mon[32714]: scrub summary: active paths [/] 2024-04-24T23:45:56.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:56 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:56 smithi107 ceph-mon[29118]: scrub queued for path: / 2024-04-24T23:45:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:56 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:56 smithi107 ceph-mon[29118]: scrub summary: active paths [/] 2024-04-24T23:45:56.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:56 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:45:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:57 smithi195 ceph-mon[33019]: pgmap v387: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-04-24T23:45:57.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:57 smithi170 ceph-mon[32714]: pgmap v387: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-04-24T23:45:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:57 smithi107 ceph-mon[29118]: pgmap v387: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-04-24T23:45:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:45:59 smithi195 ceph-mon[33019]: pgmap v388: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-24T23:45:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:45:59 smithi170 ceph-mon[32714]: pgmap v388: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-24T23:45:59.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:45:59 smithi107 ceph-mon[29118]: pgmap v388: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-24T23:46:00.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:00 smithi195 ceph-mon[33019]: scrub summary: idle 2024-04-24T23:46:00.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:00 smithi170 ceph-mon[32714]: scrub summary: idle 2024-04-24T23:46:00.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:00 smithi107 ceph-mon[29118]: scrub summary: idle 2024-04-24T23:46:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:01 smithi195 ceph-mon[33019]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-24T23:46:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:01 smithi170 ceph-mon[32714]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-24T23:46:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:01 smithi107 ceph-mon[29118]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-24T23:46:03.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:03 smithi195 ceph-mon[33019]: pgmap v390: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:03 smithi170 ceph-mon[32714]: pgmap v390: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:03.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:03 smithi107 ceph-mon[29118]: pgmap v390: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:05 smithi195 ceph-mon[33019]: pgmap v391: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:05 smithi170 ceph-mon[32714]: pgmap v391: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:05.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:05 smithi107 ceph-mon[29118]: pgmap v391: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:07 smithi195 ceph-mon[33019]: pgmap v392: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-24T23:46:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:07 smithi170 ceph-mon[32714]: pgmap v392: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-24T23:46:07.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:07 smithi107 ceph-mon[29118]: pgmap v392: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-24T23:46:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:09 smithi195 ceph-mon[33019]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:09 smithi170 ceph-mon[32714]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:09 smithi107 ceph-mon[29118]: pgmap v393: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:46:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:11 smithi195 ceph-mon[33019]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:11 smithi170 ceph-mon[32714]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:11.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:11 smithi107 ceph-mon[29118]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:13 smithi195 ceph-mon[33019]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:13 smithi170 ceph-mon[32714]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:13.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:13 smithi107 ceph-mon[29118]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-24T23:46:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:15 smithi195 ceph-mon[33019]: pgmap v396: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:15 smithi170 ceph-mon[32714]: pgmap v396: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:15.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:15 smithi107 ceph-mon[29118]: pgmap v396: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:17 smithi195 ceph-mon[33019]: pgmap v397: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:17.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:17 smithi170 ceph-mon[32714]: pgmap v397: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:17.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:17 smithi107 ceph-mon[29118]: pgmap v397: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:19 smithi195 ceph-mon[33019]: pgmap v398: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:19 smithi170 ceph-mon[32714]: pgmap v398: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:19.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:19 smithi107 ceph-mon[29118]: pgmap v398: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:21.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:21 smithi195 ceph-mon[33019]: pgmap v399: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:21 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:46:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:21 smithi170 ceph-mon[32714]: pgmap v399: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:21 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:46:21.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:21 smithi107 ceph-mon[29118]: pgmap v399: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:21.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:21 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:46:22.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: pgmap v400: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:46:22.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:22 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: pgmap v400: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:46:22.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:46:22.813 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:22 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:22.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: pgmap v400: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-24T23:46:22.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:46:22.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:22 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:46:23.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:23.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:23.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:46:24.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:24 smithi107 ceph-mon[29118]: pgmap v401: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:24.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:46:24.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:46:24.812 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:24 smithi195 ceph-mon[33019]: pgmap v401: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:24.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:24 smithi170 ceph-mon[32714]: pgmap v401: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:46:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 -- 172.21.15.107:0/1101532022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 msgr2=0x7f3f50101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/1101532022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50101cf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c02a410 comp rx=0 tx=0).stop 2024-04-24T23:46:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 -- 172.21.15.107:0/1101532022 shutdown_connections 2024-04-24T23:46:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/1101532022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f50108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/1101532022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50101cf0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/1101532022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 -- 172.21.15.107:0/1101532022 >> 172.21.15.107:0/1101532022 conn(0x7f3f500fbc30 msgr2=0x7f3f500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 -- 172.21.15.107:0/1101532022 shutdown_connections 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.909+0000 7f3f56cb3640 1 -- 172.21.15.107:0/1101532022 wait complete. 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 Processor -- start 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 -- start start 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f5019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5010d420 con 0x7f3f50101870 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5010d2a0 con 0x7f3f50100680 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f56cb3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5010d5a0 con 0x7f3f50102230 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55004/0 (socket says 172.21.15.107:55004) 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 -- 172.21.15.107:0/3334204903 learned_addr learned my addr 172.21.15.107:0/3334204903 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 -- 172.21.15.107:0/3334204903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 msgr2=0x7f3f5019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f55250640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f5019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f5019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 -- 172.21.15.107:0/3334204903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 msgr2=0x7f3f50197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f4ffff640 1 -- 172.21.15.107:0/3334204903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5019f6d0 con 0x7f3f50101870 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f55250640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f5019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.910+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009230 tx=0x7f3f3c031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f4dffb640 1 -- 172.21.15.107:0/3334204903 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c03f030 con 0x7f3f50101870 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f50109300 con 0x7f3f50101870 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f4dffb640 1 -- 172.21.15.107:0/3334204903 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c003ad0 con 0x7f3f50101870 2024-04-24T23:46:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f4dffb640 1 -- 172.21.15.107:0/3334204903 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c035070 con 0x7f3f50101870 2024-04-24T23:46:24.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.911+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f50109840 con 0x7f3f50101870 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 Processor -- start 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 -- start start 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098bf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098c4da0 0x562f098c7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562f098236b0 con 0x562f098bfc30 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562f097cb880 con 0x562f098c4da0 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f599bf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562f0981f670 con 0x562f098bf240 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f54a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f54a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55020/0 (socket says 172.21.15.107:55020) 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f54a4f640 1 -- 172.21.15.107:0/2406573207 learned_addr learned my addr 172.21.15.107:0/2406573207 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.914+0000 7f3f55250640 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 -- 172.21.15.107:0/2406573207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 msgr2=0x562f098bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 -- 172.21.15.107:0/2406573207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098c4da0 msgr2=0x562f098c7230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098c4da0 0x562f098c7230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 -- 172.21.15.107:0/2406573207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f096017d0 con 0x562f098bfc30 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400bd40 tx=0x7f3f4400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c09a7d78bc9165 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f3affd640 1 -- 172.21.15.107:0/2406573207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44011640 con 0x562f098bfc30 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f3affd640 1 -- 172.21.15.107:0/2406573207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4400be30 con 0x562f098bfc30 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f3affd640 1 -- 172.21.15.107:0/2406573207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44010b40 con 0x562f098bfc30 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.915+0000 7f3f4dffb640 1 -- 172.21.15.107:0/3334204903 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3f3c042030 con 0x7f3f50101870 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f4dffb640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 0x7f3f400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 -- 172.21.15.107:0/2406573207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 msgr2=0x562f098c4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400bd40 tx=0x7f3f4400f730 comp rx=0 tx=0).stop 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 -- 172.21.15.107:0/2406573207 shutdown_connections 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098c4da0 0x562f098c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098bfc30 0x562f098c4730 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 --2- 172.21.15.107:0/2406573207 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 -- 172.21.15.107:0/2406573207 >> 172.21.15.107:0/2406573207 conn(0x562f098ba8b0 msgr2=0x562f098bacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f4dffb640 1 -- 172.21.15.107:0/3334204903 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3f3c0ef8e0 con 0x7f3f50101870 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f55250640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 0x7f3f400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 -- 172.21.15.107:0/2406573207 shutdown_connections 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.916+0000 7f3f599bf740 1 -- 172.21.15.107:0/2406573207 wait complete. 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.917+0000 7f3f599bf740 1 -- start start 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.917+0000 7f3f55250640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 0x7f3f400a50d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x562f098c0490 tx=0x7f3f48009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.917+0000 7f3f599bf740 1 Processor -- start 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.917+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.917+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f599bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562f098c7d50 con 0x562f098c4da0 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f599bf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562f097def50 con 0x562f098bfc30 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f599bf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562f098c7ed0 con 0x562f098bf240 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f55250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f55250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55032/0 (socket says 172.21.15.107:55032) 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f55250640 1 -- 172.21.15.107:0/369221467 learned_addr learned my addr 172.21.15.107:0/369221467 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f599bf740 10 client.0 _ll_register_callbacks cb 0x562f0981a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 -- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 msgr2=0x562f098d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 -- 172.21.15.107:0/369221467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 msgr2=0x562f098d94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 -- 172.21.15.107:0/369221467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f095373f0 con 0x562f098bf240 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f55250640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d1f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x562f098bb820 tx=0x7f3f44010440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b75148b915b879b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.918+0000 7f3f2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.195:3300/0 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4402b060 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mon.1 v2:172.21.15.195:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3f440108c0 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4400be30 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f44010b40 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f599bf740 10 client.25216 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f094f70f0 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mon.1 v2:172.21.15.195:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4400be30 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.919+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mon.1 v2:172.21.15.195:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f44004920 con 0x562f098bf240 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 10 client.25216 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 10 client.25216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 20 client.25216 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f098dfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 4 client.25216 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f09830bf0 con 0x562f098dd990 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f098dfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f098dfd80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be40e72cb511aa5a server_cookie=6872b86ce58ac9e4 in_seq=0 out_seq=0 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.920+0000 7f3f2ffff640 10 client.25216 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f2ffff640 1 -- 172.21.15.107:0/369221467 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f09830bf0 con 0x562f098dd990 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f2ffff640 10 client.25216 handle_command_reply: tid=0 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 client.25216 shutdown 2024-04-24T23:46:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 msgr2=0x562f098d1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d1f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x562f098bb820 tx=0x7f3f44010440 comp rx=0 tx=0).stop 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 shutdown_connections 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f098dfd80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f2ffff640 0 client.25216 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098d94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 --2- 172.21.15.107:0/369221467 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d1f40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.921+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 >> 172.21.15.107:0/369221467 conn(0x562f098ba8b0 msgr2=0x562f098b9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.922+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 shutdown_connections 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.922+0000 7f3f599bf740 1 -- 172.21.15.107:0/369221467 wait complete. 2024-04-24T23:46:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.922+0000 7f3f599bf740 20 client.25216 trim_cache size 0 max 16384 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 Processor -- start 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 -- start start 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a8370 con 0x562f098c4da0 2024-04-24T23:46:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a9eb0 con 0x562f098bfc30 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f599bf740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a8a40 con 0x562f098bf240 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.956+0000 7f3f55250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:56868/0 (socket says 172.21.15.107:56868) 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 -- 172.21.15.107:0/1526836480 learned_addr learned my addr 172.21.15.107:0/1526836480 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 -- 172.21.15.107:0/1526836480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 msgr2=0x562f098c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098c04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 -- 172.21.15.107:0/1526836480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 msgr2=0x562f098bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098bdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 -- 172.21.15.107:0/1526836480 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f098c1770 con 0x562f098bfc30 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=915a6e35ca8bde40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f3a7fc640 1 -- 172.21.15.107:0/1526836480 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0c49e0 con 0x562f098bfc30 2024-04-24T23:46:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f3a7fc640 1 -- 172.21.15.107:0/1526836480 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c0b9e10 con 0x562f098bfc30 2024-04-24T23:46:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.957+0000 7f3f3a7fc640 1 -- 172.21.15.107:0/1526836480 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c03bc70 con 0x562f098bfc30 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 -- 172.21.15.107:0/1526836480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 msgr2=0x562f098d9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c02a410 comp rx=0 tx=0).stop 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 -- 172.21.15.107:0/1526836480 shutdown_connections 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f098bdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d9480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 --2- 172.21.15.107:0/1526836480 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 -- 172.21.15.107:0/1526836480 >> 172.21.15.107:0/1526836480 conn(0x562f098ba8b0 msgr2=0x562f098bd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.958+0000 7f3f599bf740 1 -- 172.21.15.107:0/1526836480 shutdown_connections 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 -- 172.21.15.107:0/1526836480 wait complete. 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 -- start start 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 Processor -- start 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.959+0000 7f3f599bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f55250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f55250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55044/0 (socket says 172.21.15.107:55044) 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f54a4f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f54a4f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:49154/0 (socket says 172.21.15.107:49154) 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f54a4f640 1 -- 172.21.15.107:0/306476938 learned_addr learned my addr 172.21.15.107:0/306476938 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a97e0 con 0x562f098c4da0 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a9110 con 0x562f098bfc30 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a6350 con 0x562f098bf240 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f599bf740 10 client.0 _ll_register_callbacks cb 0x7f3f44015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 -- 172.21.15.107:0/306476938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 msgr2=0x562f098d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 -- 172.21.15.107:0/306476938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 msgr2=0x562f09941de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f09942320 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f55250640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f4ffff640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d6750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x562f098d1470 tx=0x7f3f3c0b8450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fafa4b26a0f5207 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.960+0000 7f3f2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c03bc70 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f3c0b9e10 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3f3c09abe0 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c03be10 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 10 client.25295 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f099434b0 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c0b9e10 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f3c0a1020 con 0x562f098bfc30 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 10 client.25295 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 10 client.25295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 20 client.25295 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f09945cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 4 client.25295 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.961+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f098c0ba0 con 0x562f098dd990 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f09945cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f2ffff640 10 client.25295 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f54a4f640 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f09945cb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a4ab698d54ad5 server_cookie=13c9abe5eb42b86f in_seq=0 out_seq=0 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f2ffff640 1 -- 172.21.15.107:0/306476938 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562f098c0ba0 con 0x562f098dd990 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f2ffff640 10 client.25295 handle_command_reply: tid=0 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f599bf740 1 client.25295 shutdown 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.962+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 msgr2=0x562f098d6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d6750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x562f098d1470 tx=0x7f3f3c0b8450 comp rx=0 tx=0).stop 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 shutdown_connections 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x562f098dd990 0x562f09945cb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f098c4da0 0x562f09941de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f098bfc30 0x562f098d6750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 --2- 172.21.15.107:0/306476938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f098bf240 0x562f098d6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f2ffff640 0 client.25295 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 >> 172.21.15.107:0/306476938 conn(0x562f098ba8b0 msgr2=0x562f098b9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 shutdown_connections 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 1 -- 172.21.15.107:0/306476938 wait complete. 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.963+0000 7f3f599bf740 20 client.25295 trim_cache size 0 max 16384 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-04-24T23:46:24.968 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-04-24T23:46:24.969 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:46:24.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 msgr2=0x7f3f400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 0x7f3f400a50d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x562f098c0490 tx=0x7f3f48009500 comp rx=0 tx=0).stop 2024-04-24T23:46:24.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 msgr2=0x7f3f50197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009230 tx=0x7f3f3c031b60 comp rx=0 tx=0).stop 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 shutdown_connections 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3f400a2c10 0x7f3f400a50d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3f50102230 0x7f3f5019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f50101870 0x7f3f50197ff0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 --2- 172.21.15.107:0/3334204903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f50100680 0x7f3f50197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 >> 172.21.15.107:0/3334204903 conn(0x7f3f500fbc30 msgr2=0x7f3f50104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 shutdown_connections 2024-04-24T23:46:24.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:24.964+0000 7f3f56cb3640 1 -- 172.21.15.107:0/3334204903 wait complete. 2024-04-24T23:46:24.978 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T23:46:24.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.080+0000 7f0724d4f640 1 -- 172.21.15.107:0/3512945842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07201029a0 msgr2=0x7f0720102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.080+0000 7f0724d4f640 1 --2- 172.21.15.107:0/3512945842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07201029a0 0x7f0720102e20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0714009900 tx=0x7f071402a410 comp rx=0 tx=0).stop 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 -- 172.21.15.107:0/3512945842 shutdown_connections 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 --2- 172.21.15.107:0/3512945842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0720103360 0x7f0720109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 --2- 172.21.15.107:0/3512945842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07201029a0 0x7f0720102e20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 --2- 172.21.15.107:0/3512945842 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201017b0 0x7f0720101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 -- 172.21.15.107:0/3512945842 >> 172.21.15.107:0/3512945842 conn(0x7f07200fcd20 msgr2=0x7f07200ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 -- 172.21.15.107:0/3512945842 shutdown_connections 2024-04-24T23:46:25.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.081+0000 7f0724d4f640 1 -- 172.21.15.107:0/3512945842 wait complete. 2024-04-24T23:46:25.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 Processor -- start 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 -- start start 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07201017b0 0x7f0720197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201029a0 0x7f0720198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f072010d430 con 0x7f0720103360 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f072010d2b0 con 0x7f07201017b0 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f0724d4f640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f072010d5b0 con 0x7f07201029a0 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55062/0 (socket says 172.21.15.107:55062) 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071ed9d640 1 -- 172.21.15.107:0/1140950178 learned_addr learned my addr 172.21.15.107:0/1140950178 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071e59c640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07201017b0 0x7f0720197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071dd9b640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201029a0 0x7f0720198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.082+0000 7f071ed9d640 1 -- 172.21.15.107:0/1140950178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201029a0 msgr2=0x7f0720198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f071ed9d640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201029a0 0x7f0720198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f071ed9d640 1 -- 172.21.15.107:0/1140950178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07201017b0 msgr2=0x7f0720197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f071ed9d640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07201017b0 0x7f0720197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f071ed9d640 1 -- 172.21.15.107:0/1140950178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072019f6f0 con 0x7f0720103360 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f071ed9d640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f071800eaa0 tx=0x7f071800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071800cdb0 con 0x7f0720103360 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072019f9e0 con 0x7f0720103360 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718005020 con 0x7f0720103360 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180116b0 con 0x7f0720103360 2024-04-24T23:46:25.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.083+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072019ff00 con 0x7f0720103360 2024-04-24T23:46:25.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.084+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0720068560 con 0x7f0720103360 2024-04-24T23:46:25.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.085+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0718005560 con 0x7f0720103360 2024-04-24T23:46:25.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.086+0000 7f070f7fe640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 0x7f07080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.086+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f07180cf710 con 0x7f0720103360 2024-04-24T23:46:25.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.086+0000 7f071e59c640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 0x7f07080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.087+0000 7f071e59c640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 0x7f07080a5050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0710005fd0 tx=0x7f0710005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.095+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0718016030 con 0x7f0720103360 2024-04-24T23:46:25.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.534+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0720108af0 con 0x7f0720103360 2024-04-24T23:46:25.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.535+0000 7f070f7fe640 1 -- 172.21.15.107:0/1140950178 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7f071809c2a0 con 0x7f0720103360 2024-04-24T23:46:25.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:46:25.535 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:46:25.536 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:46:25.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 msgr2=0x7f07080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 0x7f07080a5050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0710005fd0 tx=0x7f0710005850 comp rx=0 tx=0).stop 2024-04-24T23:46:25.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 msgr2=0x7f072019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f071800eaa0 tx=0x7f071800ef60 comp rx=0 tx=0).stop 2024-04-24T23:46:25.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 shutdown_connections 2024-04-24T23:46:25.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f07080a2b90 0x7f07080a5050 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0720103360 0x7f072019f080 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f07201029a0 0x7f0720198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 --2- 172.21.15.107:0/1140950178 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07201017b0 0x7f0720197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 >> 172.21.15.107:0/1140950178 conn(0x7f07200fcd20 msgr2=0x7f0720105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 shutdown_connections 2024-04-24T23:46:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.538+0000 7f0724d4f640 1 -- 172.21.15.107:0/1140950178 wait complete. 2024-04-24T23:46:25.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-24T23:46:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 -- 172.21.15.107:0/1988763358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4101790 msgr2=0x7f96a4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 --2- 172.21.15.107:0/1988763358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4101790 0x7f96a4101bb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9694009900 tx=0x7f969402a410 comp rx=0 tx=0).stop 2024-04-24T23:46:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 -- 172.21.15.107:0/1988763358 shutdown_connections 2024-04-24T23:46:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 --2- 172.21.15.107:0/1988763358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4103340 0x7f96a4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 --2- 172.21.15.107:0/1988763358 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4102980 0x7f96a4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 --2- 172.21.15.107:0/1988763358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4101790 0x7f96a4101bb0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.652+0000 7f96aa415640 1 -- 172.21.15.107:0/1988763358 >> 172.21.15.107:0/1988763358 conn(0x7f96a40fcd20 msgr2=0x7f96a40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 -- 172.21.15.107:0/1988763358 shutdown_connections 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 -- 172.21.15.107:0/1988763358 wait complete. 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 Processor -- start 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 -- start start 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4101790 0x7f96a40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.653+0000 7f96aa415640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96aa415640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 0x7f96a4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96aa415640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d400 con 0x7f96a4102980 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96aa415640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d280 con 0x7f96a4101790 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96aa415640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f96a410d580 con 0x7f96a4103340 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55092/0 (socket says 172.21.15.107:55092) 2024-04-24T23:46:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 -- 172.21.15.107:0/2819198713 learned_addr learned my addr 172.21.15.107:0/2819198713 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a3fff640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4101790 0x7f96a40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a89b2640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 0x7f96a4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 -- 172.21.15.107:0/2819198713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 msgr2=0x7f96a4074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 0x7f96a4074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 -- 172.21.15.107:0/2819198713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4101790 msgr2=0x7f96a40754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4101790 0x7f96a40754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 -- 172.21.15.107:0/2819198713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a4074900 con 0x7f96a4102980 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a89b2640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 0x7f96a4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f969bfff640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c970 tx=0x7f968c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a1ffb640 1 -- 172.21.15.107:0/2819198713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c007bb0 con 0x7f96a4102980 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a41a1c00 con 0x7f96a4102980 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a1ffb640 1 -- 172.21.15.107:0/2819198713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968c007d50 con 0x7f96a4102980 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.654+0000 7f96a1ffb640 1 -- 172.21.15.107:0/2819198713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c00f450 con 0x7f96a4102980 2024-04-24T23:46:25.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.655+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a41a2150 con 0x7f96a4102980 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96a1ffb640 1 -- 172.21.15.107:0/2819198713 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f968c005020 con 0x7f96a4102980 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 Processor -- start 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 -- start start 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592cc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556f59228670 con 0x556f592c9d90 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556f591d0970 con 0x556f592c4c20 2024-04-24T23:46:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96ad201740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x556f59224630 con 0x556f592c4230 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96a89b2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55106/0 (socket says 172.21.15.107:55106) 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f969bfff640 1 -- 172.21.15.107:0/4188800512 learned_addr learned my addr 172.21.15.107:0/4188800512 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96a89b2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:49198/0 (socket says 172.21.15.107:49198) 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96a3fff640 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.658+0000 7f96a1ffb640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 0x7f96900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 -- 172.21.15.107:0/4188800512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 msgr2=0x556f592c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 -- 172.21.15.107:0/4188800512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 msgr2=0x556f592cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 -- 172.21.15.107:0/4188800512 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f591f4400 con 0x556f592c4230 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a1ffb640 1 -- 172.21.15.107:0/2819198713 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f968c0cf270 con 0x7f96a4102980 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 0x7f96900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f96a89b2640 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f969c005fd0 tx=0x7f969c00f8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f7f03837984a6a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f969a7fc640 1 -- 172.21.15.107:0/4188800512 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c021070 con 0x556f592c4230 2024-04-24T23:46:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f969a7fc640 1 -- 172.21.15.107:0/4188800512 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f969c01a3c0 con 0x556f592c4230 2024-04-24T23:46:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.659+0000 7f969a7fc640 1 -- 172.21.15.107:0/4188800512 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c019560 con 0x556f592c4230 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- 172.21.15.107:0/4188800512 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 msgr2=0x556f592c4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96a89b2640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 0x7f96900a5050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f969c01e000 tx=0x7f969c01b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f969c005fd0 tx=0x7f969c00f8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- 172.21.15.107:0/4188800512 shutdown_connections 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 --2- 172.21.15.107:0/4188800512 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4230 0x556f592c4650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- 172.21.15.107:0/4188800512 >> 172.21.15.107:0/4188800512 conn(0x556f592bf8a0 msgr2=0x556f592bfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- 172.21.15.107:0/4188800512 shutdown_connections 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- 172.21.15.107:0/4188800512 wait complete. 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.660+0000 7f96ad201740 1 -- start start 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 Processor -- start 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4c20 0x556f592ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556f592ccd40 con 0x556f592c9d90 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556f591e4050 con 0x556f592c4230 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x556f592ccec0 con 0x556f592c4c20 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96ad201740 10 client.0 _ll_register_callbacks cb 0x556f58f3bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55120/0 (socket says 172.21.15.107:55120) 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 -- 172.21.15.107:0/3083008217 learned_addr learned my addr 172.21.15.107:0/3083008217 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f969bfff640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4c20 0x556f592ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 -- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4c20 msgr2=0x556f592ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4c20 0x556f592ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 -- 172.21.15.107:0/3083008217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 msgr2=0x556f592ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.661+0000 7f96a89b2640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96a89b2640 1 -- 172.21.15.107:0/3083008217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f591b5dc0 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96a89b2640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f969c01f040 tx=0x7f969c0098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9617d4af95a718 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96837fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c032060 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f969c01bcb0 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f969c01a3c0 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f969c021040 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96ad201740 10 client.15888 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f58f3c3f0 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.662+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f969c01a3c0 con 0x556f592c9d90 2024-04-24T23:46:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f969c019040 con 0x556f592c9d90 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 10 client.15888 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 10 client.15888 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 20 client.15888 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f592d9dc0 0x556f592dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 4 client.15888 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f59235bb0 con 0x556f592d9dc0 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f592d9dc0 0x556f592dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.663+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f592d9dc0 0x556f592dc1b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c432527876d5927 server_cookie=9c30048a38369b4a in_seq=0 out_seq=0 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96837fe640 10 client.15888 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96837fe640 1 -- 172.21.15.107:0/3083008217 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f59235bb0 con 0x556f592d9dc0 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96837fe640 10 client.15888 handle_command_reply: tid=0 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96ad201740 1 client.15888 shutdown 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 msgr2=0x556f592d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.664+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f969c01f040 tx=0x7f969c0098a0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 shutdown_connections 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f592d9dc0 0x556f592dc1b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c9d90 0x556f592d58d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c4c20 0x556f592ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96837fe640 0 client.15888 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 --2- 172.21.15.107:0/3083008217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 >> 172.21.15.107:0/3083008217 conn(0x556f592bf8a0 msgr2=0x556f592beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 shutdown_connections 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 1 -- 172.21.15.107:0/3083008217 wait complete. 2024-04-24T23:46:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.665+0000 7f96ad201740 20 client.15888 trim_cache size 0 max 16384 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 Processor -- start 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 -- start start 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41ac730 con 0x556f592c4c20 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41ae270 con 0x556f592c4230 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96ad201740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41ace00 con 0x556f592c9d90 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f969bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55130/0 (socket says 172.21.15.107:55130) 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f969bfff640 1 -- 172.21.15.107:0/356120925 learned_addr learned my addr 172.21.15.107:0/356120925 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.678+0000 7f96a3fff640 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 -- 172.21.15.107:0/356120925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 msgr2=0x556f592d5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 -- 172.21.15.107:0/356120925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 msgr2=0x556f592c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592c2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 -- 172.21.15.107:0/356120925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f590067d0 con 0x556f592c4c20 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f969bfff640 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f968c0a3e40 tx=0x7f968c097a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d0dbe1bae49bb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f9699ffb640 1 -- 172.21.15.107:0/356120925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c0cce60 con 0x556f592c4c20 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f9699ffb640 1 -- 172.21.15.107:0/356120925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968c09be00 con 0x556f592c4c20 2024-04-24T23:46:25.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.679+0000 7f9699ffb640 1 -- 172.21.15.107:0/356120925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c01d040 con 0x556f592c4c20 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- 172.21.15.107:0/356120925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 msgr2=0x556f592c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f968c0a3e40 tx=0x7f968c097a20 comp rx=0 tx=0).stop 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- 172.21.15.107:0/356120925 shutdown_connections 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4c20 0x556f592c31d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 --2- 172.21.15.107:0/356120925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4230 0x556f592c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- 172.21.15.107:0/356120925 >> 172.21.15.107:0/356120925 conn(0x556f592bf8a0 msgr2=0x556f592c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- 172.21.15.107:0/356120925 shutdown_connections 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- 172.21.15.107:0/356120925 wait complete. 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.680+0000 7f96ad201740 1 -- start start 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 Processor -- start 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55136/0 (socket says 172.21.15.107:55136) 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 -- 172.21.15.107:0/3440292798 learned_addr learned my addr 172.21.15.107:0/3440292798 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41adba0 con 0x556f592c4230 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41ad4d0 con 0x556f592c4c20 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41aa710 con 0x556f592c9d90 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96ad201740 10 client.0 _ll_register_callbacks cb 0x7f969c01f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f969bfff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a89b2640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 -- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 msgr2=0x556f592d4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 -- 172.21.15.107:0/3440292798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 msgr2=0x556f592d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592d3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a3fff640 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f592d4e70 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f969bfff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.681+0000 7f96a89b2640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96a3fff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9694005ab0 tx=0x7f969402aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff47005d9fba0d29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694035040 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9694004340 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f968c09be00 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9694054050 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96ad201740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f593405f0 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968c09be00 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f96940317d0 con 0x556f592c4230 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.682+0000 7f96ad201740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 10 client.15894 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 20 client.15894 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f59341020 0x556f59343440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 4 client.15894 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f5922ff00 con 0x556f59341020 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f969bfff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f59341020 0x556f59343440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f969bfff640 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f59341020 0x556f59343440 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=29c01adedf488f96 server_cookie=df4967521b1a8ae0 in_seq=0 out_seq=0 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96837fe640 10 client.15894 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96837fe640 1 -- 172.21.15.107:0/3440292798 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f5922ff00 con 0x556f59341020 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96837fe640 10 client.15894 handle_command_reply: tid=0 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.683+0000 7f96ad201740 1 client.15894 shutdown 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 msgr2=0x556f592d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9694005ab0 tx=0x7f969402aba0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 shutdown_connections 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x556f59341020 0x556f59343440 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556f592c9d90 0x556f592d4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556f592c4c20 0x556f592d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96837fe640 0 client.15894 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 --2- 172.21.15.107:0/3440292798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556f592c4230 0x556f592d3330 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 >> 172.21.15.107:0/3440292798 conn(0x556f592bf8a0 msgr2=0x556f592c2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 shutdown_connections 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 1 -- 172.21.15.107:0/3440292798 wait complete. 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.684+0000 7f96ad201740 20 client.15894 trim_cache size 0 max 16384 2024-04-24T23:46:25.689 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 msgr2=0x7f96900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 0x7f96900a5050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f969c01e000 tx=0x7f969c01b000 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 msgr2=0x7f96a4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c970 tx=0x7f968c00ce30 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 shutdown_connections 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f96900a2b90 0x7f96900a5050 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f96a4103340 0x7f96a4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96a4102980 0x7f96a4073b30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 --2- 172.21.15.107:0/2819198713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a4101790 0x7f96a40754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 >> 172.21.15.107:0/2819198713 conn(0x7f96a40fcd20 msgr2=0x7f96a4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 shutdown_connections 2024-04-24T23:46:25.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.685+0000 7f96aa415640 1 -- 172.21.15.107:0/2819198713 wait complete. 2024-04-24T23:46:25.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 -- 172.21.15.107:0/1431067858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18101870 msgr2=0x7eff18101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 --2- 172.21.15.107:0/1431067858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18101870 0x7eff18101cf0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7eff08009900 tx=0x7eff0802a410 comp rx=0 tx=0).stop 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 -- 172.21.15.107:0/1431067858 shutdown_connections 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 --2- 172.21.15.107:0/1431067858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18102230 0x7eff18108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 --2- 172.21.15.107:0/1431067858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18101870 0x7eff18101cf0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 --2- 172.21.15.107:0/1431067858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18100680 0x7eff18100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 -- 172.21.15.107:0/1431067858 >> 172.21.15.107:0/1431067858 conn(0x7eff180fbbf0 msgr2=0x7eff180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 -- 172.21.15.107:0/1431067858 shutdown_connections 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.797+0000 7eff20412640 1 -- 172.21.15.107:0/1431067858 wait complete. 2024-04-24T23:46:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 Processor -- start 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 -- start start 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18101870 0x7eff1819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 0x7eff1819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff1810c300 con 0x7eff18100680 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff1810c180 con 0x7eff18101870 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff20412640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7eff1810c480 con 0x7eff18102230 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55158/0 (socket says 172.21.15.107:55158) 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 -- 172.21.15.107:0/3420868932 learned_addr learned my addr 172.21.15.107:0/3420868932 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e9af640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 0x7eff1819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18101870 0x7eff1819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 -- 172.21.15.107:0/3420868932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 msgr2=0x7eff1819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 0x7eff1819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 -- 172.21.15.107:0/3420868932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18101870 msgr2=0x7eff1819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18101870 0x7eff1819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e1ae640 1 -- 172.21.15.107:0/3420868932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1819f780 con 0x7eff18100680 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.798+0000 7eff1e9af640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 0x7eff1819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7eff1400d950 tx=0x7eff1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff0f7fe640 1 -- 172.21.15.107:0/3420868932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14017070 con 0x7eff18100680 2024-04-24T23:46:25.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1819fa10 con 0x7eff18100680 2024-04-24T23:46:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff0f7fe640 1 -- 172.21.15.107:0/3420868932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff14009c30 con 0x7eff18100680 2024-04-24T23:46:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff0f7fe640 1 -- 172.21.15.107:0/3420868932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1400f600 con 0x7eff18100680 2024-04-24T23:46:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.799+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1819ff30 con 0x7eff18100680 2024-04-24T23:46:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff0f7fe640 1 -- 172.21.15.107:0/3420868932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7eff1401a070 con 0x7eff18100680 2024-04-24T23:46:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 Processor -- start 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 -- start start 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950a3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950a8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950aac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c894ffe670 con 0x55c8950a5f30 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55c894fa6970 con 0x55c8950a8860 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff231fb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55c894ffa630 con 0x55c8950a3100 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950a8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.802+0000 7eff1e9af640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950a8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55174/0 (socket says 172.21.15.107:55174) 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1e1ae640 1 -- 172.21.15.107:0/331825996 learned_addr learned my addr 172.21.15.107:0/331825996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff0f7fe640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 0x7eff040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 -- 172.21.15.107:0/331825996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 msgr2=0x55c8950a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 -- 172.21.15.107:0/331825996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 msgr2=0x55c8950a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950a8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 -- 172.21.15.107:0/331825996 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c894fca400 con 0x55c8950a8860 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff0f7fe640 1 -- 172.21.15.107:0/3420868932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7eff140cefc0 con 0x7eff18100680 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950aac50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff080314e0 tx=0x7eff08005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d483ec58f01e756 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.803+0000 7eff1e9af640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 0x7eff040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff0cbfe640 1 -- 172.21.15.107:0/331825996 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0803f070 con 0x55c8950a8860 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff0cbfe640 1 -- 172.21.15.107:0/331825996 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff08004030 con 0x55c8950a8860 2024-04-24T23:46:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff0cbfe640 1 -- 172.21.15.107:0/331825996 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08035070 con 0x55c8950a8860 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 -- 172.21.15.107:0/331825996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 msgr2=0x55c8950aac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950aac50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff080314e0 tx=0x7eff08005e60 comp rx=0 tx=0).stop 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff1e9af640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 0x7eff040a50d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7eff10005fd0 tx=0x7eff10004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 -- 172.21.15.107:0/331825996 shutdown_connections 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950aac50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950a8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 --2- 172.21.15.107:0/331825996 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.804+0000 7eff231fb740 1 -- 172.21.15.107:0/331825996 >> 172.21.15.107:0/331825996 conn(0x55c895095820 msgr2=0x55c895095c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- 172.21.15.107:0/331825996 shutdown_connections 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- 172.21.15.107:0/331825996 wait complete. 2024-04-24T23:46:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- start start 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 Processor -- start 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950acb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c8950ab630 con 0x55c8950a5f30 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55c894fba050 con 0x55c8950a8860 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55c8950ab7b0 con 0x55c8950a3100 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.805+0000 7eff231fb740 10 client.0 _ll_register_callbacks cb 0x55c894d11be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1d9ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:56990/0 (socket says 172.21.15.107:56990) 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1d9ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55190/0 (socket says 172.21.15.107:55190) 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 -- 172.21.15.107:0/2381682127 learned_addr learned my addr 172.21.15.107:0/2381682127 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 -- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 msgr2=0x55c8950acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 -- 172.21.15.107:0/2381682127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 msgr2=0x55c8950ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 -- 172.21.15.107:0/2381682127 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c894f8bdc0 con 0x55c8950a8860 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff1e9af640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff100129c0 tx=0x7eff10004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7acc46db07505282 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1001dd70 con 0x55c8950a8860 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff08004030 con 0x55c8950a8860 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.806+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7eff10027470 con 0x55c8950a8860 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7eff10030050 con 0x55c8950a8860 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 10 client.25331 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c894d123f0 con 0x55c8950a8860 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff08004030 con 0x55c8950a8860 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7eff1001e720 con 0x55c8950a8860 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 10 client.25331 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 10 client.25331 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 20 client.25331 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c8950b89d0 0x55c8950bae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 4 client.25331 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c89500bbb0 con 0x55c8950b89d0 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.807+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c8950b89d0 0x55c8950bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.808+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c8950b89d0 0x55c8950bae10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=17fcfa13e318e00a server_cookie=4ef7904c82683336 in_seq=0 out_seq=0 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.808+0000 7eff01ffb640 10 client.25331 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.808+0000 7eff01ffb640 1 -- 172.21.15.107:0/2381682127 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c89500bbb0 con 0x55c8950b89d0 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.808+0000 7eff01ffb640 10 client.25331 handle_command_reply: tid=0 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.808+0000 7eff231fb740 1 client.25331 shutdown 2024-04-24T23:46:25.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 msgr2=0x55c8950b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff100129c0 tx=0x7eff10004ab0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 shutdown_connections 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c8950b89d0 0x55c8950bae10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b4080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 --2- 172.21.15.107:0/2381682127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a3100 0x55c8950acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 >> 172.21.15.107:0/2381682127 conn(0x55c895095820 msgr2=0x55c895094b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff01ffb640 0 client.25331 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 shutdown_connections 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 1 -- 172.21.15.107:0/2381682127 wait complete. 2024-04-24T23:46:25.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.809+0000 7eff231fb740 20 client.25331 trim_cache size 0 max 16384 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 Processor -- start 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 -- start start 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a3100 0x55c8950aa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a8860 0x55c8950a2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55194/0 (socket says 172.21.15.107:55194) 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 -- 172.21.15.107:0/1001497264 learned_addr learned my addr 172.21.15.107:0/1001497264 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a82d0 con 0x55c8950a5f30 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a9e10 con 0x55c8950a3100 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a89a0 con 0x55c8950a8860 2024-04-24T23:46:25.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a3100 0x55c8950aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1e9af640 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a8860 0x55c8950a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 -- 172.21.15.107:0/1001497264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a8860 msgr2=0x55c8950a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a8860 0x55c8950a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 -- 172.21.15.107:0/1001497264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a3100 msgr2=0x55c8950aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.822+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a3100 0x55c8950aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.823+0000 7eff1d9ad640 1 -- 172.21.15.107:0/1001497264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c894ddc7d0 con 0x55c8950a5f30 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.823+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7eff08009900 tx=0x7eff08032db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd17c842d45e021d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.823+0000 7eff03fff640 1 -- 172.21.15.107:0/1001497264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08035070 con 0x55c8950a5f30 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.823+0000 7eff03fff640 1 -- 172.21.15.107:0/1001497264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff080318b0 con 0x55c8950a5f30 2024-04-24T23:46:25.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.823+0000 7eff03fff640 1 -- 172.21.15.107:0/1001497264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0803f040 con 0x55c8950a5f30 2024-04-24T23:46:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 msgr2=0x55c8950b4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7eff08009900 tx=0x7eff08032db0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 shutdown_connections 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a8860 0x55c8950a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a5f30 0x55c8950b4040 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- 172.21.15.107:0/1001497264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a3100 0x55c8950aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 >> 172.21.15.107:0/1001497264 conn(0x55c895095820 msgr2=0x55c8950a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 shutdown_connections 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- 172.21.15.107:0/1001497264 wait complete. 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- start start 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 Processor -- start 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a5f30 0x55c8950b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a9740 con 0x55c8950a3100 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a9070 con 0x55c8950a8860 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7eff181a62b0 con 0x55c8950a5f30 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff231fb740 10 client.0 _ll_register_callbacks cb 0x7eff10009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff1e1ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55208/0 (socket says 172.21.15.107:55208) 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.824+0000 7eff1e1ae640 1 -- 172.21.15.107:0/1060682331 learned_addr learned my addr 172.21.15.107:0/1060682331 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a5f30 0x55c8950b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e9af640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 -- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a5f30 msgr2=0x55c8950b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a5f30 0x55c8950b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 -- 172.21.15.107:0/1060682331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 msgr2=0x55c8950b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 -- 172.21.15.107:0/1060682331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8950b3100 con 0x55c8950a3100 2024-04-24T23:46:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e9af640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff1e1ae640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7eff1409b380 tx=0x7eff14097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87ba4d0b917b3e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14021040 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7eff14097a60 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff080318b0 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7eff14009de0 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff231fb740 10 client.15912 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.825+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c8950b4470 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff080318b0 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7eff140d2020 con 0x55c8950a3100 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 10 client.15912 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 10 client.15912 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 20 client.15912 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c89511f730 0x55c895121b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 4 client.15912 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c895005f00 con 0x55c89511f730 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c89511f730 0x55c895121b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff01ffb640 10 client.15912 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.826+0000 7eff1d9ad640 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c89511f730 0x55c895121b50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a34e9f4a703f9e17 server_cookie=18968d57a5d93e50 in_seq=0 out_seq=0 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff01ffb640 1 -- 172.21.15.107:0/1060682331 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c895005f00 con 0x55c89511f730 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff01ffb640 10 client.15912 handle_command_reply: tid=0 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 client.15912 shutdown 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 msgr2=0x55c8950b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7eff1409b380 tx=0x7eff14097040 comp rx=0 tx=0).stop 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 shutdown_connections 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x55c89511f730 0x55c895121b50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff01ffb640 0 client.15912 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c8950a8860 0x55c8950b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55c8950a5f30 0x55c8950b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 --2- 172.21.15.107:0/1060682331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c8950a3100 0x55c8950b15c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 >> 172.21.15.107:0/1060682331 conn(0x55c895095820 msgr2=0x55c8950a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 shutdown_connections 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 1 -- 172.21.15.107:0/1060682331 wait complete. 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff231fb740 20 client.15912 trim_cache size 0 max 16384 2024-04-24T23:46:25.833 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 msgr2=0x7eff040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 0x7eff040a50d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7eff10005fd0 tx=0x7eff10004a40 comp rx=0 tx=0).stop 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 msgr2=0x7eff18069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.827+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7eff1400d950 tx=0x7eff1400de10 comp rx=0 tx=0).stop 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 shutdown_connections 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7eff040a2c10 0x7eff040a50d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7eff18102230 0x7eff1819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff18101870 0x7eff1819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 --2- 172.21.15.107:0/3420868932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff18100680 0x7eff18069440 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 >> 172.21.15.107:0/3420868932 conn(0x7eff180fbbf0 msgr2=0x7eff18104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 shutdown_connections 2024-04-24T23:46:25.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.828+0000 7eff20412640 1 -- 172.21.15.107:0/3420868932 wait complete. 2024-04-24T23:46:25.840 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:25 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1140950178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:25.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T23:46:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.945+0000 7f1ce1329640 1 -- 172.21.15.107:0/3862564956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc1012f0 msgr2=0x7f1cdc101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.945+0000 7f1ce1329640 1 --2- 172.21.15.107:0/3862564956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc101770 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009ca0 tx=0x7f1cd402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 -- 172.21.15.107:0/3862564956 shutdown_connections 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 --2- 172.21.15.107:0/3862564956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc101770 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 --2- 172.21.15.107:0/3862564956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 0x7f1cdc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 --2- 172.21.15.107:0/3862564956 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 -- 172.21.15.107:0/3862564956 >> 172.21.15.107:0/3862564956 conn(0x7f1cdc0fbc30 msgr2=0x7f1cdc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 -- 172.21.15.107:0/3862564956 shutdown_connections 2024-04-24T23:46:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 -- 172.21.15.107:0/3862564956 wait complete. 2024-04-24T23:46:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.946+0000 7f1ce1329640 1 Processor -- start 2024-04-24T23:46:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 -- start start 2024-04-24T23:46:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 0x7f1cdc19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc10f5f0 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc10f470 con 0x7f1cdc100950 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1ce1329640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc10f770 con 0x7f1cdc1012f0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55224/0 (socket says 172.21.15.107:55224) 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 -- 172.21.15.107:0/2217881950 learned_addr learned my addr 172.21.15.107:0/2217881950 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 0x7f1cdc19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 -- 172.21.15.107:0/2217881950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc1012f0 msgr2=0x7f1cdc1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 -- 172.21.15.107:0/2217881950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 msgr2=0x7f1cdc19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 0x7f1cdc19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.947+0000 7f1cdaffd640 1 -- 172.21.15.107:0/2217881950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cdc1a1880 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9f0 tx=0x7f1ccc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1cc3fff640 1 -- 172.21.15.107:0/2217881950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00cd80 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cdc1a1b70 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1cc3fff640 1 -- 172.21.15.107:0/2217881950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ccc005500 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1cc3fff640 1 -- 172.21.15.107:0/2217881950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc010690 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.948+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cdc1a2070 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 Processor -- start 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cc3fff640 1 -- 172.21.15.107:0/2217881950 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1ccc0046c0 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 -- start start 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c54775650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c5477a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559c546d9670 con 0x559c5477ad90 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559c54681970 con 0x559c54775230 2024-04-24T23:46:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1ce403a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559c546d5630 con 0x559c54775c20 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55240/0 (socket says 172.21.15.107:55240) 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3496093192 learned_addr learned my addr 172.21.15.107:0/3496093192 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c54775650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c5477a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3496093192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 msgr2=0x559c5477a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c5477a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3496093192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 msgr2=0x559c54775650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c54775650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3496093192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c546a5400 con 0x559c5477ad90 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.951+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c5477a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cc3fff640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 0x7f1cc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009cf0 tx=0x7f1cd0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c5c509f500d2ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 0x7f1cc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cc3fff640 1 -- 172.21.15.107:0/2217881950 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1ccc0cf8e0 con 0x7f1cdc0ffff0 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cc17fa640 1 -- 172.21.15.107:0/3496093192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0004230 con 0x559c5477ad90 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cc17fa640 1 -- 172.21.15.107:0/3496093192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd00043d0 con 0x559c5477ad90 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.952+0000 7f1cc17fa640 1 -- 172.21.15.107:0/3496093192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0019920 con 0x559c5477ad90 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- 172.21.15.107:0/3496093192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 msgr2=0x559c5477d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 0x7f1cc40a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4005e30 tx=0x7f1cd403d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009cf0 tx=0x7f1cd0011810 comp rx=0 tx=0).stop 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- 172.21.15.107:0/3496093192 shutdown_connections 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c5477d220 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c5477a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3496093192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c54775650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- 172.21.15.107:0/3496093192 >> 172.21.15.107:0/3496093192 conn(0x559c547708a0 msgr2=0x559c54770ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- 172.21.15.107:0/3496093192 shutdown_connections 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- 172.21.15.107:0/3496093192 wait complete. 2024-04-24T23:46:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 -- start start 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.953+0000 7f1ce403a740 1 Processor -- start 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c5477f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c5477ad90 0x559c54786870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559c5477dd40 con 0x559c54775c20 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559c54695050 con 0x559c54775230 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x559c5477dec0 con 0x559c5477ad90 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1ce403a740 10 client.0 _ll_register_callbacks cb 0x559c543ecbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55256/0 (socket says 172.21.15.107:55256) 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3742206894 learned_addr learned my addr 172.21.15.107:0/3742206894 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c5477ad90 0x559c54786870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c5477f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3742206894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c5477ad90 msgr2=0x559c54786870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c5477ad90 0x559c54786870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 msgr2=0x559c5477f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c5477f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cda7fc640 1 -- 172.21.15.107:0/3742206894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c54666dc0 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.954+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c5477f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0004630 tx=0x7f1cd00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b46377dd31892 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0027070 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1cd0005240 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd00043d0 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1cd0033050 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1ce403a740 10 client.15924 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559c543ed3f0 con 0x559c54775c20 2024-04-24T23:46:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.955+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd00043d0 con 0x559c54775c20 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1cd0004cc0 con 0x559c54775c20 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 10 client.15924 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 10 client.15924 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 20 client.15924 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c5478b090 0x559c5478d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 4 client.15924 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c546e6bb0 con 0x559c5478b090 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c5478b090 0x559c5478d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1cbeff5640 10 client.15924 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.956+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c5478b090 0x559c5478d510 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a53f85f645ec51cd server_cookie=7ab95c8ed3f3b477 in_seq=0 out_seq=0 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1cbeff5640 1 -- 172.21.15.107:0/3742206894 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c546e6bb0 con 0x559c5478b090 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1cbeff5640 10 client.15924 handle_command_reply: tid=0 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 client.15924 shutdown 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 msgr2=0x559c5477f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0004630 tx=0x7f1cd00038a0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 shutdown_connections 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c5478b090 0x559c5478d510 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c5477ad90 0x559c54786870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c5477f830 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3742206894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c5477f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 >> 172.21.15.107:0/3742206894 conn(0x559c547708a0 msgr2=0x559c5476fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.957+0000 7f1cbeff5640 0 client.15924 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.958+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 shutdown_connections 2024-04-24T23:46:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.958+0000 7f1ce403a740 1 -- 172.21.15.107:0/3742206894 wait complete. 2024-04-24T23:46:25.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.958+0000 7f1ce403a740 20 client.15924 trim_cache size 0 max 16384 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 Processor -- start 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 -- start start 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c547766c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c54786830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1ac630 con 0x559c5477ad90 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1ae170 con 0x559c54775230 2024-04-24T23:46:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1ce403a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1acd00 con 0x559c54775c20 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1cdb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1cdb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55272/0 (socket says 172.21.15.107:55272) 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1cdb7fe640 1 -- 172.21.15.107:0/3804609593 learned_addr learned my addr 172.21.15.107:0/3804609593 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.976+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c547766c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c54786830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 -- 172.21.15.107:0/3804609593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 msgr2=0x559c54786830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c54786830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 -- 172.21.15.107:0/3804609593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 msgr2=0x559c547766c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c547766c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 -- 172.21.15.107:0/3804609593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c544b77d0 con 0x559c5477ad90 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c547766c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cdb7fe640 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4035000 tx=0x7f1cd4041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e3a37e86775525 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cc0ff9640 1 -- 172.21.15.107:0/3804609593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd402fce0 con 0x559c5477ad90 2024-04-24T23:46:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cc0ff9640 1 -- 172.21.15.107:0/3804609593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd4057070 con 0x559c5477ad90 2024-04-24T23:46:25.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.977+0000 7f1cc0ff9640 1 -- 172.21.15.107:0/3804609593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4052750 con 0x559c5477ad90 2024-04-24T23:46:25.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 -- 172.21.15.107:0/3804609593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 msgr2=0x559c54776c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4035000 tx=0x7f1cd4041000 comp rx=0 tx=0).stop 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 -- 172.21.15.107:0/3804609593 shutdown_connections 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c5477ad90 0x559c54776c00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775c20 0x559c54786830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 --2- 172.21.15.107:0/3804609593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c54775230 0x559c547766c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 -- 172.21.15.107:0/3804609593 >> 172.21.15.107:0/3804609593 conn(0x559c547708a0 msgr2=0x559c5477f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 -- 172.21.15.107:0/3804609593 shutdown_connections 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.978+0000 7f1ce403a740 1 -- 172.21.15.107:0/3804609593 wait complete. 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 -- start start 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 Processor -- start 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775230 0x559c54783690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c5477ad90 0x559c547ef680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1adaa0 con 0x559c54775c20 2024-04-24T23:46:25.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1ad3d0 con 0x559c5477ad90 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc1aa610 con 0x559c54775230 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55274/0 (socket says 172.21.15.107:55274) 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1cda7fc640 1 -- 172.21.15.107:0/1494961722 learned_addr learned my addr 172.21.15.107:0/1494961722 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775230 0x559c54783690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.979+0000 7f1ce403a740 10 client.0 _ll_register_callbacks cb 0x7f1ccc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 -- 172.21.15.107:0/1494961722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775230 msgr2=0x559c54783690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775230 0x559c54783690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 -- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c5477ad90 msgr2=0x559c547ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c5477ad90 0x559c547ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 -- 172.21.15.107:0/1494961722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c547efbc0 con 0x559c54775c20 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cda7fc640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009cf0 tx=0x7f1cd0009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced7abfe275e1614 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:25.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0027040 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1cd0019b00 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd4057070 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1cd0009ea0 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.980+0000 7f1ce403a740 10 client.15930 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559c547f0df0 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd4057070 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1cd0004660 con 0x559c54775c20 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 10 client.15930 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 10 client.15930 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 20 client.15930 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c547f17c0 0x559c547f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 4 client.15930 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:25.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c546e0f00 con 0x559c547f17c0 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.981+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c547f17c0 0x559c547f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1cbeff5640 10 client.15930 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1cdaffd640 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c547f17c0 0x559c547f3be0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=721f79a198588a7d server_cookie=6a0ed97bc196c53b in_seq=0 out_seq=0 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1cbeff5640 1 -- 172.21.15.107:0/1494961722 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c546e0f00 con 0x559c547f17c0 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1cbeff5640 10 client.15930 handle_command_reply: tid=0 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 client.15930 shutdown 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 msgr2=0x559c54783bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009cf0 tx=0x7f1cd0009be0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 shutdown_connections 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x559c547f17c0 0x559c547f3be0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c5477ad90 0x559c547ef680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559c54775c20 0x559c54783bd0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 --2- 172.21.15.107:0/1494961722 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x559c54775230 0x559c54783690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 >> 172.21.15.107:0/1494961722 conn(0x559c547708a0 msgr2=0x559c5476f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1cbeff5640 0 client.15930 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 shutdown_connections 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 1 -- 172.21.15.107:0/1494961722 wait complete. 2024-04-24T23:46:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.982+0000 7f1ce403a740 20 client.15930 trim_cache size 0 max 16384 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 msgr2=0x7f1cc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 0x7f1cc40a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4005e30 tx=0x7f1cd403d040 comp rx=0 tx=0).stop 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 msgr2=0x7f1cdc199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9f0 tx=0x7f1ccc00edf0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 shutdown_connections 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1cc40a2b90 0x7f1cc40a5050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1cdc1012f0 0x7f1cdc1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cdc100950 0x7f1cdc19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 --2- 172.21.15.107:0/2217881950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cdc0ffff0 0x7f1cdc199cc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 >> 172.21.15.107:0/2217881950 conn(0x7f1cdc0fbc30 msgr2=0x7f1cdc0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 shutdown_connections 2024-04-24T23:46:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:25.983+0000 7f1ce1329640 1 -- 172.21.15.107:0/2217881950 wait complete. 2024-04-24T23:46:25.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T23:46:26.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:25 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1140950178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:26.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:25 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1140950178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 -- 172.21.15.107:0/1689924598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8101870 msgr2=0x7fbdd8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 --2- 172.21.15.107:0/1689924598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8101870 0x7fbdd8101cf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009900 tx=0x7fbdcc02a410 comp rx=0 tx=0).stop 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 -- 172.21.15.107:0/1689924598 shutdown_connections 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 --2- 172.21.15.107:0/1689924598 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8102230 0x7fbdd8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 --2- 172.21.15.107:0/1689924598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8101870 0x7fbdd8101cf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 --2- 172.21.15.107:0/1689924598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8100680 0x7fbdd8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 -- 172.21.15.107:0/1689924598 >> 172.21.15.107:0/1689924598 conn(0x7fbdd80fbbf0 msgr2=0x7fbdd80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 -- 172.21.15.107:0/1689924598 shutdown_connections 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.098+0000 7fbddea1b640 1 -- 172.21.15.107:0/1689924598 wait complete. 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 Processor -- start 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 -- start start 2024-04-24T23:46:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8101870 0x7fbdd8196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 0x7fbdd819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd810c2e0 con 0x7fbdd8100680 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd810c160 con 0x7fbdd8102230 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddea1b640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd810c460 con 0x7fbdd8101870 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55288/0 (socket says 172.21.15.107:55288) 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2196923397 learned_addr learned my addr 172.21.15.107:0/2196923397 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8101870 0x7fbdd8196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 0x7fbdd819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2196923397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8101870 msgr2=0x7fbdd8196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8101870 0x7fbdd8196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2196923397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 msgr2=0x7fbdd819df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 0x7fbdd819df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.099+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2196923397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd819e5a0 con 0x7fbdd8100680 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 0x7fbdd819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bea0 tx=0x7fbdc800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbdd57fa640 1 -- 172.21.15.107:0/2196923397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8016070 con 0x7fbdd8100680 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd8068930 con 0x7fbdd8100680 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbdd57fa640 1 -- 172.21.15.107:0/2196923397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdc8014070 con 0x7fbdd8100680 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbdd57fa640 1 -- 172.21.15.107:0/2196923397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc801b6c0 con 0x7fbdd8100680 2024-04-24T23:46:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.100+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd8068ea0 con 0x7fbdd8100680 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.102+0000 7fbdd57fa640 1 -- 172.21.15.107:0/2196923397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbdc8010560 con 0x7fbdd8100680 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 Processor -- start 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 -- start start 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c333690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c340b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b53c294670 con 0x55b53c3301b0 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b53c23c970 con 0x55b53c330ba0 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbde17fd740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b53c290630 con 0x55b53c333d00 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55300/0 (socket says 172.21.15.107:55300) 2024-04-24T23:46:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 -- 172.21.15.107:0/146719127 learned_addr learned my addr 172.21.15.107:0/146719127 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c340b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbdd57fa640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 0x7fbdc00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 -- 172.21.15.107:0/146719127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 msgr2=0x55b53c340b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c340b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 -- 172.21.15.107:0/146719127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 msgr2=0x55b53c333690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c333690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbdd57fa640 1 -- 172.21.15.107:0/2196923397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbdc80d1230 con 0x7fbdd8100680 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c333690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.103+0000 7fbddcfb8640 1 -- 172.21.15.107:0/146719127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b53c260400 con 0x55b53c3301b0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 0x7fbdc00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c333690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0008f60 tx=0x7fbdd0008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f0023750846d65 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbdbe7fc640 1 -- 172.21.15.107:0/146719127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd001a8e0 con 0x55b53c3301b0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbdbe7fc640 1 -- 172.21.15.107:0/146719127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd0019410 con 0x55b53c3301b0 2024-04-24T23:46:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbdbe7fc640 1 -- 172.21.15.107:0/146719127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0018600 con 0x55b53c3301b0 2024-04-24T23:46:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.104+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 0x7fbdc00a5150 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbdd001e000 tx=0x7fbdd001f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- 172.21.15.107:0/146719127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 msgr2=0x55b53c3305d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0008f60 tx=0x7fbdd0008f90 comp rx=0 tx=0).stop 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- 172.21.15.107:0/146719127 shutdown_connections 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c340b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c333690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 --2- 172.21.15.107:0/146719127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c3305d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- 172.21.15.107:0/146719127 >> 172.21.15.107:0/146719127 conn(0x55b53c32b8a0 msgr2=0x55b53c32bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- 172.21.15.107:0/146719127 shutdown_connections 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- 172.21.15.107:0/146719127 wait complete. 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.105+0000 7fbde17fd740 1 -- start start 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 Processor -- start 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c343080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c34a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b53c341570 con 0x55b53c3301b0 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b53c250050 con 0x55b53c330ba0 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55302/0 (socket says 172.21.15.107:55302) 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2799199894 learned_addr learned my addr 172.21.15.107:0/2799199894 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55b53c3416f0 con 0x55b53c333d00 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbde17fd740 10 client.0 _ll_register_callbacks cb 0x55b53bfa7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c34a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c343080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 msgr2=0x55b53c34a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c34a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2799199894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 msgr2=0x55b53c343080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c343080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2799199894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53c221dc0 con 0x55b53c3301b0 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c343080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.106+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80a5da0 tx=0x7fbdc8010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb0fe9b13738982 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc809cfc0 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdd0019410 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbdc8010850 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbdc80880b0 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 10 client.15942 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b53bfa83f0 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd0019410 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbdc8023030 con 0x55b53c3301b0 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 10 client.15942 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 10 client.15942 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 20 client.15942 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c34ed00 0x55b53c351190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 4 client.15942 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.107+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b53c2a1bb0 con 0x55b53c34ed00 2024-04-24T23:46:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.108+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c34ed00 0x55b53c351190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.108+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c34ed00 0x55b53c351190 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2aca11b7457a7316 server_cookie=924c4b6f54aa3e57 in_seq=0 out_seq=0 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.108+0000 7fbdb77fe640 10 client.15942 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbdb77fe640 1 -- 172.21.15.107:0/2799199894 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b53c2a1bb0 con 0x55b53c34ed00 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbdb77fe640 10 client.15942 handle_command_reply: tid=0 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 client.15942 shutdown 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 msgr2=0x55b53c342b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80a5da0 tx=0x7fbdc8010c50 comp rx=0 tx=0).stop 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 shutdown_connections 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c333d00 0x55b53c34a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c34ed00 0x55b53c351190 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c330ba0 0x55b53c343080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2799199894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342b40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 >> 172.21.15.107:0/2799199894 conn(0x55b53c32b8a0 msgr2=0x55b53c32ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbdb77fe640 0 client.15942 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 shutdown_connections 2024-04-24T23:46:26.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 1 -- 172.21.15.107:0/2799199894 wait complete. 2024-04-24T23:46:26.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.109+0000 7fbde17fd740 20 client.15942 trim_cache size 0 max 16384 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 Processor -- start 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 -- start start 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 0x55b53c34a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c340760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a8380 con 0x55b53c3301b0 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a9ec0 con 0x55b53c333d00 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbde17fd740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a8a50 con 0x55b53c330ba0 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55316/0 (socket says 172.21.15.107:55316) 2024-04-24T23:46:26.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2610692332 learned_addr learned my addr 172.21.15.107:0/2610692332 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 0x55b53c34a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c340760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2610692332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 msgr2=0x55b53c34a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 0x55b53c34a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2610692332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 msgr2=0x55b53c340760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c340760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd7fff640 1 -- 172.21.15.107:0/2610692332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b53c0727d0 con 0x55b53c3301b0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.123+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 0x55b53c34a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.124+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8099ac0 tx=0x7fbdc800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cff52fb695e1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.124+0000 7fbdbdffb640 1 -- 172.21.15.107:0/2610692332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc809d430 con 0x55b53c3301b0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.124+0000 7fbdbdffb640 1 -- 172.21.15.107:0/2610692332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdc8009c30 con 0x55b53c3301b0 2024-04-24T23:46:26.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.124+0000 7fbdbdffb640 1 -- 172.21.15.107:0/2610692332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc801e120 con 0x55b53c3301b0 2024-04-24T23:46:26.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- 172.21.15.107:0/2610692332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 msgr2=0x55b53c342fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8099ac0 tx=0x7fbdc800b6e0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- 172.21.15.107:0/2610692332 shutdown_connections 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c340760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c330ba0 0x55b53c34a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 --2- 172.21.15.107:0/2610692332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c3301b0 0x55b53c342fe0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- 172.21.15.107:0/2610692332 >> 172.21.15.107:0/2610692332 conn(0x55b53c32b8a0 msgr2=0x55b53c32a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- 172.21.15.107:0/2610692332 shutdown_connections 2024-04-24T23:46:26.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- 172.21.15.107:0/2610692332 wait complete. 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 -- start start 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.125+0000 7fbde17fd740 1 Processor -- start 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c3301b0 0x55b53c347680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c348bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a97f0 con 0x55b53c330ba0 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a9120 con 0x55b53c333d00 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55324/0 (socket says 172.21.15.107:55324) 2024-04-24T23:46:26.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 -- 172.21.15.107:0/202911745 learned_addr learned my addr 172.21.15.107:0/202911745 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81a6360 con 0x55b53c3301b0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c3301b0 0x55b53c347680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbddcfb8640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c348bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 10 client.0 _ll_register_callbacks cb 0x7fbdc80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 -- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c3301b0 msgr2=0x55b53c347680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c3301b0 0x55b53c347680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 -- 172.21.15.107:0/202911745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 msgr2=0x55b53c348bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c348bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 -- 172.21.15.107:0/202911745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53c3491c0 con 0x55b53c330ba0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdc8009c30 con 0x55b53c330ba0 2024-04-24T23:46:26.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdd77fe640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0314e0 tx=0x7fbdcc032ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a3063b63509685 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.126+0000 7fbdb77fe640 10 client.15948 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbdb77fe640 1 -- 172.21.15.107:0/202911745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc035040 con 0x55b53c330ba0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbdb77fe640 1 -- 172.21.15.107:0/202911745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbdcc004240 con 0x55b53c330ba0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbdb77fe640 1 -- 172.21.15.107:0/202911745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbdcc004480 con 0x55b53c330ba0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 10 client.15948 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55b53c34a420 con 0x55b53c330ba0 2024-04-24T23:46:26.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbdb77fe640 1 -- 172.21.15.107:0/202911745 <== mon.0 v2:172.21.15.107:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbdcc031850 con 0x55b53c330ba0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 10 client.15948 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 10 client.15948 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 20 client.15948 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c3b59c0 0x55b53c3b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 4 client.15948 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.127+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b53c29bf00 con 0x55b53c3b59c0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c3b59c0 0x55b53c3b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbdd7fff640 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c3b59c0 0x55b53c3b7de0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47633d95eb34df2f server_cookie=431d5fb26e770ae4 in_seq=0 out_seq=0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbdb77fe640 10 client.15948 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbdb77fe640 1 -- 172.21.15.107:0/202911745 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b53c29bf00 con 0x55b53c3b59c0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbdb77fe640 10 client.15948 handle_command_reply: tid=0 2024-04-24T23:46:26.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbde17fd740 1 client.15948 shutdown 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 msgr2=0x55b53c347bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0314e0 tx=0x7fbdcc032ef0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 shutdown_connections 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.128+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x55b53c3b59c0 0x55b53c3b7de0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbdb77fe640 0 client.15948 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b53c333d00 0x55b53c348bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b53c330ba0 0x55b53c347bc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 --2- 172.21.15.107:0/202911745 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55b53c3301b0 0x55b53c347680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 >> 172.21.15.107:0/202911745 conn(0x55b53c32b8a0 msgr2=0x55b53c32a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 shutdown_connections 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 1 -- 172.21.15.107:0/202911745 wait complete. 2024-04-24T23:46:26.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbde17fd740 20 client.15948 trim_cache size 0 max 16384 2024-04-24T23:46:26.138 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 msgr2=0x7fbdc00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 0x7fbdc00a5150 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbdd001e000 tx=0x7fbdd001f000 comp rx=0 tx=0).stop 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 msgr2=0x7fbdd8196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bea0 tx=0x7fbdc800bf90 comp rx=0 tx=0).stop 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 shutdown_connections 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fbdc00a2c90 0x7fbdc00a5150 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdd8102230 0x7fbdd819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fbdd8101870 0x7fbdd8196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 --2- 172.21.15.107:0/2196923397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd8100680 0x7fbdd8196980 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.129+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 >> 172.21.15.107:0/2196923397 conn(0x7fbdd80fbbf0 msgr2=0x7fbdd8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.130+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 shutdown_connections 2024-04-24T23:46:26.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.130+0000 7fbddea1b640 1 -- 172.21.15.107:0/2196923397 wait complete. 2024-04-24T23:46:26.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 -- 172.21.15.107:0/3506132087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 msgr2=0x7fea4c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 --2- 172.21.15.107:0/3506132087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c108ac0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fea44009ca0 tx=0x7fea4402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 -- 172.21.15.107:0/3506132087 shutdown_connections 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 --2- 172.21.15.107:0/3506132087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c108ac0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 --2- 172.21.15.107:0/3506132087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c101870 0x7fea4c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 --2- 172.21.15.107:0/3506132087 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c100680 0x7fea4c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 -- 172.21.15.107:0/3506132087 >> 172.21.15.107:0/3506132087 conn(0x7fea4c0fbbf0 msgr2=0x7fea4c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 -- 172.21.15.107:0/3506132087 shutdown_connections 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.243+0000 7fea50c1c640 1 -- 172.21.15.107:0/3506132087 wait complete. 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 Processor -- start 2024-04-24T23:46:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 -- start start 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 0x7fea4c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10d3e0 con 0x7fea4c102230 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10d260 con 0x7fea4c100680 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea50c1c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10d560 con 0x7fea4c101870 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:49400/0 (socket says 172.21.15.107:49400) 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55338/0 (socket says 172.21.15.107:55338) 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.244+0000 7fea49d9b640 1 -- 172.21.15.107:0/357113894 learned_addr learned my addr 172.21.15.107:0/357113894 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 -- 172.21.15.107:0/357113894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 msgr2=0x7fea4c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea4a59c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 0x7fea4c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 0x7fea4c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 -- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 msgr2=0x7fea4c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 -- 172.21.15.107:0/357113894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c19f670 con 0x7fea4c101870 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea4a59c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 0x7fea4c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea49d9b640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea4000c890 tx=0x7fea4000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea3b7fe640 1 -- 172.21.15.107:0/357113894 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40004220 con 0x7fea4c101870 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c19f960 con 0x7fea4c101870 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea3b7fe640 1 -- 172.21.15.107:0/357113894 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea400043c0 con 0x7fea4c101870 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea3b7fe640 1 -- 172.21.15.107:0/357113894 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea400049e0 con 0x7fea4c101870 2024-04-24T23:46:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.245+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c19ff00 con 0x7fea4c101870 2024-04-24T23:46:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.248+0000 7fea3b7fe640 1 -- 172.21.15.107:0/357113894 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fea40004c40 con 0x7fea4c101870 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 Processor -- start 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 -- start start 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 0x55e06f193720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f196220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e06f0f2670 con 0x55e06f18ec20 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e06f09a970 con 0x55e06f193d90 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea53a08740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e06f0ee630 con 0x55e06f18e230 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea3b7fe640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 0x7fea340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:49416/0 (socket says 172.21.15.107:49416) 2024-04-24T23:46:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.249+0000 7fea4ad9d640 1 -- 172.21.15.107:0/3236915220 learned_addr learned my addr 172.21.15.107:0/3236915220 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea3b7fe640 1 -- 172.21.15.107:0/357113894 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fea400cf510 con 0x7fea4c101870 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea49d9b640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f196220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 0x7fea340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4a59c640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 0x55e06f193720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 -- 172.21.15.107:0/3236915220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 msgr2=0x55e06f196220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f196220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 -- 172.21.15.107:0/3236915220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 msgr2=0x55e06f193720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 0x55e06f193720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 -- 172.21.15.107:0/3236915220 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06f0be400 con 0x55e06f18e230 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4a59c640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 0x55e06f193720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 0x7fea340a5050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fea4402f540 tx=0x7fea44036000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.250+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea44005e30 tx=0x7fea4402f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f5653193b8f7ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea38ff9640 1 -- 172.21.15.107:0/3236915220 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4404d070 con 0x55e06f18e230 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea38ff9640 1 -- 172.21.15.107:0/3236915220 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea44036070 con 0x55e06f18e230 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea38ff9640 1 -- 172.21.15.107:0/3236915220 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44049a80 con 0x55e06f18e230 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 -- 172.21.15.107:0/3236915220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 msgr2=0x55e06f18e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea44005e30 tx=0x7fea4402f690 comp rx=0 tx=0).stop 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 -- 172.21.15.107:0/3236915220 shutdown_connections 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f196220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18ec20 0x55e06f193720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 --2- 172.21.15.107:0/3236915220 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18e650 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 -- 172.21.15.107:0/3236915220 >> 172.21.15.107:0/3236915220 conn(0x55e06f1898a0 msgr2=0x55e06f189ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 -- 172.21.15.107:0/3236915220 shutdown_connections 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.251+0000 7fea53a08740 1 -- 172.21.15.107:0/3236915220 wait complete. 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 -- start start 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 Processor -- start 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f198720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e06f196b60 con 0x55e06f193d90 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e06f0ae050 con 0x55e06f18ec20 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e06f196ce0 con 0x55e06f18e230 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea53a08740 10 client.0 _ll_register_callbacks cb 0x55e06ee05be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55344/0 (socket says 172.21.15.107:55344) 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.252+0000 7fea4ad9d640 1 -- 172.21.15.107:0/938170337 learned_addr learned my addr 172.21.15.107:0/938170337 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea49d9b640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4a59c640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 -- 172.21.15.107:0/938170337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 msgr2=0x55e06f1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 -- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 msgr2=0x55e06f198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 -- 172.21.15.107:0/938170337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06f07fdc0 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fea44031ae0 tx=0x7fea4402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cc6aa01da80fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44057070 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fea44004d40 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea44036070 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea4404d050 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea53a08740 10 client.15960 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.253+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e06ee063f0 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea44031880 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea440050c0 con 0x55e06f193d90 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 10 client.15960 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 20 client.15960 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f1a3e20 0x55e06f1a6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 4 client.15960 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fea44036070 con 0x55e06f1a3e20 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.254+0000 7fea4a59c640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f1a3e20 0x55e06f1a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea31ffb640 10 client.15960 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea4a59c640 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f1a3e20 0x55e06f1a6260 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bb0411132402835 server_cookie=e62aba54e5b6d827 in_seq=0 out_seq=0 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea31ffb640 1 -- 172.21.15.107:0/938170337 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fea44036070 con 0x55e06f1a3e20 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea31ffb640 10 client.15960 handle_command_reply: tid=0 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 client.15960 shutdown 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 msgr2=0x55e06f19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fea44031ae0 tx=0x7fea4402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 shutdown_connections 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f1a3e20 0x55e06f1a6260 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f19f760 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea31ffb640 0 client.15960 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 --2- 172.21.15.107:0/938170337 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.255+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 >> 172.21.15.107:0/938170337 conn(0x55e06f1898a0 msgr2=0x55e06f188b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.256+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 shutdown_connections 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.256+0000 7fea53a08740 1 -- 172.21.15.107:0/938170337 wait complete. 2024-04-24T23:46:26.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.256+0000 7fea53a08740 20 client.15960 trim_cache size 0 max 16384 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 Processor -- start 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 -- start start 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f19f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1aa3f0 con 0x55e06f193d90 2024-04-24T23:46:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1abf30 con 0x55e06f18ec20 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea53a08740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1aaac0 con 0x55e06f18e230 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55346/0 (socket says 172.21.15.107:55346) 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 -- 172.21.15.107:0/540086252 learned_addr learned my addr 172.21.15.107:0/540086252 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea49d9b640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4a59c640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 -- 172.21.15.107:0/540086252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 msgr2=0x55e06f18c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 -- 172.21.15.107:0/540086252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 msgr2=0x55e06f19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea4ad9d640 1 -- 172.21.15.107:0/540086252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06eed07d0 con 0x55e06f193d90 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.269+0000 7fea49d9b640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.270+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fea4402ade0 tx=0x7fea44030ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cde846636141efd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.270+0000 7fea33fff640 1 -- 172.21.15.107:0/540086252 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44031880 con 0x55e06f193d90 2024-04-24T23:46:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.270+0000 7fea33fff640 1 -- 172.21.15.107:0/540086252 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea4402f610 con 0x55e06f193d90 2024-04-24T23:46:26.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.270+0000 7fea33fff640 1 -- 172.21.15.107:0/540086252 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44051400 con 0x55e06f193d90 2024-04-24T23:46:26.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- 172.21.15.107:0/540086252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 msgr2=0x55e06f18cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fea4402ade0 tx=0x7fea44030ca0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- 172.21.15.107:0/540086252 shutdown_connections 2024-04-24T23:46:26.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f193d90 0x55e06f18cac0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f18ec20 0x55e06f19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 --2- 172.21.15.107:0/540086252 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18e230 0x55e06f18c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- 172.21.15.107:0/540086252 >> 172.21.15.107:0/540086252 conn(0x55e06f1898a0 msgr2=0x55e06f188b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- 172.21.15.107:0/540086252 shutdown_connections 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- 172.21.15.107:0/540086252 wait complete. 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 -- start start 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.271+0000 7fea53a08740 1 Processor -- start 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 0x55e06f19d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f19e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55354/0 (socket says 172.21.15.107:55354) 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 -- 172.21.15.107:0/405341712 learned_addr learned my addr 172.21.15.107:0/405341712 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1ab860 con 0x55e06f18e230 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1ab190 con 0x55e06f193d90 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1a83d0 con 0x55e06f18ec20 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea49d9b640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 0x55e06f19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 10 client.0 _ll_register_callbacks cb 0x7fea3c009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 -- 172.21.15.107:0/405341712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 msgr2=0x55e06f19d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 0x55e06f19d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 -- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 msgr2=0x55e06f19e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f19e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06f19ee40 con 0x55e06f18e230 2024-04-24T23:46:26.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea49d9b640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 0x55e06f19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4ad9d640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea4402f610 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea4a59c640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009c20 tx=0x7fea3c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdebac327f78972 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.272+0000 7fea31ffb640 10 client.15966 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea31ffb640 1 -- 172.21.15.107:0/405341712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c005240 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea31ffb640 1 -- 172.21.15.107:0/405341712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fea3c0053e0 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea31ffb640 1 -- 172.21.15.107:0/405341712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea3c0055e0 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 10 client.15966 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55e06f20a890 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea31ffb640 1 -- 172.21.15.107:0/405341712 <== mon.0 v2:172.21.15.107:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea3c022070 con 0x55e06f18e230 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 10 client.15966 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 20 client.15966 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f20b060 0x55e06f20d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 4 client.15966 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.273+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e06f0ffbb0 con 0x55e06f20b060 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea49d9b640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f20b060 0x55e06f20d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea31ffb640 10 client.15966 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea49d9b640 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f20b060 0x55e06f20d480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da5475a754a2295c server_cookie=b3acc1ef57306339 in_seq=0 out_seq=0 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea31ffb640 1 -- 172.21.15.107:0/405341712 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e06f0ffbb0 con 0x55e06f20b060 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea31ffb640 10 client.15966 handle_command_reply: tid=0 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea53a08740 1 client.15966 shutdown 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 msgr2=0x55e06f19d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009c20 tx=0x7fea3c01bf90 comp rx=0 tx=0).stop 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 shutdown_connections 2024-04-24T23:46:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.274+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e06f20b060 0x55e06f20d480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e06f193d90 0x55e06f19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea31ffb640 0 client.15966 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e06f18ec20 0x55e06f19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 --2- 172.21.15.107:0/405341712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e06f18e230 0x55e06f19d300 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 >> 172.21.15.107:0/405341712 conn(0x55e06f1898a0 msgr2=0x55e06f18f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 shutdown_connections 2024-04-24T23:46:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 1 -- 172.21.15.107:0/405341712 wait complete. 2024-04-24T23:46:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea53a08740 20 client.15966 trim_cache size 0 max 16384 2024-04-24T23:46:26.283 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 msgr2=0x7fea340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 0x7fea340a5050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fea4402f540 tx=0x7fea44036000 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 msgr2=0x7fea4c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea4000c890 tx=0x7fea4000cd50 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 shutdown_connections 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fea340a2b90 0x7fea340a5050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea4c102230 0x7fea4c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fea4c101870 0x7fea4c197f90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 --2- 172.21.15.107:0/357113894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c100680 0x7fea4c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.275+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 >> 172.21.15.107:0/357113894 conn(0x7fea4c0fbbf0 msgr2=0x7fea4c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.276+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 shutdown_connections 2024-04-24T23:46:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:26.276+0000 7fea50c1c640 1 -- 172.21.15.107:0/357113894 wait complete. 2024-04-24T23:46:26.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:26 smithi107 ceph-mon[29118]: pgmap v402: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:27.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:26 smithi195 ceph-mon[33019]: pgmap v402: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:27.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:26 smithi170 ceph-mon[32714]: pgmap v402: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:46:27.289 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T23:46:27.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:46:27.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T23:46:27.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 -- 172.21.15.107:0/3559964479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c102230 msgr2=0x7ff27c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 --2- 172.21.15.107:0/3559964479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c102230 0x7ff27c108ac0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff274009ca0 tx=0x7ff27402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 -- 172.21.15.107:0/3559964479 shutdown_connections 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 --2- 172.21.15.107:0/3559964479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c102230 0x7ff27c108ac0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 --2- 172.21.15.107:0/3559964479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 --2- 172.21.15.107:0/3559964479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c100680 0x7ff27c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 -- 172.21.15.107:0/3559964479 >> 172.21.15.107:0/3559964479 conn(0x7ff27c0fbbf0 msgr2=0x7ff27c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 -- 172.21.15.107:0/3559964479 shutdown_connections 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.394+0000 7ff283da9640 1 -- 172.21.15.107:0/3559964479 wait complete. 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 Processor -- start 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 -- start start 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 0x7ff27c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c10c360 con 0x7ff27c100680 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c10c1e0 con 0x7ff27c102230 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff283da9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c10c4e0 con 0x7ff27c101870 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.395+0000 7ff281b45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59768/0 (socket says 172.21.15.107:59768) 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 -- 172.21.15.107:0/1816792764 learned_addr learned my addr 172.21.15.107:0/1816792764 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff282346640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 0x7ff27c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281344640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 -- 172.21.15.107:0/1816792764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 msgr2=0x7ff27c196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 -- 172.21.15.107:0/1816792764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 msgr2=0x7ff27c19df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 0x7ff27c19df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 -- 172.21.15.107:0/1816792764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c19e590 con 0x7ff27c100680 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281344640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff282346640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 0x7ff27c19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff281b45640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff278007b50 tx=0x7ff27800e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff272ffd640 1 -- 172.21.15.107:0/1816792764 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278007c40 con 0x7ff27c100680 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27c068930 con 0x7ff27c100680 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff272ffd640 1 -- 172.21.15.107:0/1816792764 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff278007de0 con 0x7ff27c100680 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff272ffd640 1 -- 172.21.15.107:0/1816792764 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27800ad10 con 0x7ff27c100680 2024-04-24T23:46:27.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.396+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27c068f00 con 0x7ff27c100680 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.399+0000 7ff286b99740 1 Processor -- start 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.399+0000 7ff286b99740 1 -- start start 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492447c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447c740 0x56492447eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5649243d42c0 con 0x56492447f100 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56492437c910 con 0x564924479e10 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff286b99740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5649243d0280 con 0x56492447c740 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59780/0 (socket says 172.21.15.107:59780) 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 -- 172.21.15.107:0/1646107047 learned_addr learned my addr 172.21.15.107:0/1646107047 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff282346640 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492447c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff272ffd640 1 -- 172.21.15.107:0/1816792764 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff27800a440 con 0x7ff27c100680 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 -- 172.21.15.107:0/1646107047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447c740 msgr2=0x56492447eb30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447c740 0x56492447eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 -- 172.21.15.107:0/1646107047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 msgr2=0x56492447c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492447c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff281344640 1 -- 172.21.15.107:0/1646107047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492438dfb0 con 0x56492447f100 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.400+0000 7ff282346640 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492447c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff281344640 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a9d0 tx=0x7ff26c00ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2801da5103413ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff272ffd640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 0x7ff2680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff267fff640 1 -- 172.21.15.107:0/1646107047 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c015070 con 0x56492447f100 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff267fff640 1 -- 172.21.15.107:0/1646107047 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff26c00cc60 con 0x56492447f100 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff267fff640 1 -- 172.21.15.107:0/1646107047 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c018600 con 0x56492447f100 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff282346640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 0x7ff2680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.401+0000 7ff272ffd640 1 -- 172.21.15.107:0/1816792764 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2780cea80 con 0x7ff27c100680 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff282346640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 0x7ff2680a5050 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x56492446ff70 tx=0x7ff274031bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 -- 172.21.15.107:0/1646107047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 msgr2=0x564924481590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a9d0 tx=0x7ff26c00ae90 comp rx=0 tx=0).stop 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 -- 172.21.15.107:0/1646107047 shutdown_connections 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447f100 0x564924481590 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447c740 0x56492447eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 --2- 172.21.15.107:0/1646107047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492447c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 -- 172.21.15.107:0/1646107047 >> 172.21.15.107:0/1646107047 conn(0x564924438950 msgr2=0x56492446c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 -- 172.21.15.107:0/1646107047 shutdown_connections 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.402+0000 7ff286b99740 1 -- 172.21.15.107:0/1646107047 wait complete. 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 -- start start 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 Processor -- start 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x564924483b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492448b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5649244820a0 con 0x56492447c740 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56492438fe90 con 0x564924479e10 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x564924482220 con 0x56492447f100 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff286b99740 10 client.0 _ll_register_callbacks cb 0x5649243cb970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59786/0 (socket says 172.21.15.107:59786) 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.403+0000 7ff281344640 1 -- 172.21.15.107:0/2897242641 learned_addr learned my addr 172.21.15.107:0/2897242641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281b45640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x564924483b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff282346640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492448b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 -- 172.21.15.107:0/2897242641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 msgr2=0x56492448b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492448b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 -- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 msgr2=0x564924483b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x564924483b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 -- 172.21.15.107:0/2897242641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649241b07e0 con 0x56492447c740 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff281344640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c520 tx=0x7ff26c01a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe4ef52fea86f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff2657fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c029060 con 0x56492447c740 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff26c01a7f0 con 0x56492447c740 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff26c00cc60 con 0x56492447c740 2024-04-24T23:46:27.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff26c030050 con 0x56492447c740 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff286b99740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.404+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564924210f70 con 0x56492447c740 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff26c00cc60 con 0x56492447c740 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff26c0185d0 con 0x56492447c740 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 20 client.15978 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56492448f210 0x564924491600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 4 client.15978 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649243a85a0 con 0x56492448f210 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.405+0000 7ff281b45640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56492448f210 0x564924491600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff2657fa640 10 client.15978 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff281b45640 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56492448f210 0x564924491600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fa35576b9f508b1 server_cookie=e9b49ee37b881a1f in_seq=0 out_seq=0 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff2657fa640 1 -- 172.21.15.107:0/2897242641 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649243a85a0 con 0x56492448f210 2024-04-24T23:46:27.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff2657fa640 10 client.15978 handle_command_reply: tid=0 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff286b99740 1 client.15978 shutdown 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 msgr2=0x564924484070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.406+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c520 tx=0x7ff26c01a680 comp rx=0 tx=0).stop 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 shutdown_connections 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x56492448f210 0x564924491600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492448b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff2657fa640 0 client.15978 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x564924484070 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 --2- 172.21.15.107:0/2897242641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x564924483b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 >> 172.21.15.107:0/2897242641 conn(0x564924438950 msgr2=0x564924479690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 shutdown_connections 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 1 -- 172.21.15.107:0/2897242641 wait complete. 2024-04-24T23:46:27.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.407+0000 7ff286b99740 20 client.15978 trim_cache size 0 max 16384 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.427+0000 7ff286b99740 1 Processor -- start 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 -- start start 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492446c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492446b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a8420 con 0x56492447c740 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a9f60 con 0x564924479e10 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff286b99740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a8af0 con 0x56492447f100 2024-04-24T23:46:27.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59796/0 (socket says 172.21.15.107:59796) 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 -- 172.21.15.107:0/4215892700 learned_addr learned my addr 172.21.15.107:0/4215892700 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281b45640 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492446c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff282346640 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492446b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 -- 172.21.15.107:0/4215892700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 msgr2=0x56492446b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492446b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 -- 172.21.15.107:0/4215892700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 msgr2=0x56492446c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492446c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 -- 172.21.15.107:0/4215892700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649240a8170 con 0x56492447c740 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.428+0000 7ff281344640 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff26c018ff0 tx=0x7ff26c006e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d464742ecc8420c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff2677fe640 1 -- 172.21.15.107:0/4215892700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c029050 con 0x56492447c740 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff2677fe640 1 -- 172.21.15.107:0/4215892700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff26c006ba0 con 0x56492447c740 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff2677fe640 1 -- 172.21.15.107:0/4215892700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c015040 con 0x56492447c740 2024-04-24T23:46:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 -- 172.21.15.107:0/4215892700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 msgr2=0x56492448b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff26c018ff0 tx=0x7ff26c006e40 comp rx=0 tx=0).stop 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 -- 172.21.15.107:0/4215892700 shutdown_connections 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x56492446b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56492447c740 0x56492448b070 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 --2- 172.21.15.107:0/4215892700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924479e10 0x56492446c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.429+0000 7ff286b99740 1 -- 172.21.15.107:0/4215892700 >> 172.21.15.107:0/4215892700 conn(0x564924438950 msgr2=0x56492446c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- 172.21.15.107:0/4215892700 shutdown_connections 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- 172.21.15.107:0/4215892700 wait complete. 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- start start 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 Processor -- start 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x564924489310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924489850 0x564924489cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a9890 con 0x564924479e10 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a91c0 con 0x564924489850 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1a6400 con 0x56492447f100 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff281b45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff281b45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59812/0 (socket says 172.21.15.107:59812) 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff281b45640 1 -- 172.21.15.107:0/3460734364 learned_addr learned my addr 172.21.15.107:0/3460734364 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.430+0000 7ff286b99740 10 client.0 _ll_register_callbacks cb 0x7ff274009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff282346640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924489850 0x564924489cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281344640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x564924489310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 -- 172.21.15.107:0/3460734364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 msgr2=0x564924489310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x564924489310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 -- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924489850 msgr2=0x564924489cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924489850 0x564924489cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 -- 172.21.15.107:0/3460734364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649244f5600 con 0x564924479e10 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff281b45640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff2780cc240 tx=0x7ff27800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfb31d3011dbb1f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff2657fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278050040 con 0x564924479e10 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff27801bd90 con 0x564924479e10 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649243e1800 con 0x564924479e10 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff27809b030 con 0x564924479e10 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff286b99740 10 client.15984 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5649244f6890 con 0x564924479e10 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.431+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5649243e1800 con 0x564924479e10 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff278014070 con 0x564924479e10 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 10 client.15984 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 20 client.15984 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5649244f7190 0x5649244f9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 4 client.15984 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649243dbb50 con 0x5649244f7190 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff281344640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5649244f7190 0x5649244f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff2657fa640 10 client.15984 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.432+0000 7ff281344640 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5649244f7190 0x5649244f9580 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd2f9ab28bf505f server_cookie=966cfbec406d3db5 in_seq=0 out_seq=0 2024-04-24T23:46:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff2657fa640 1 -- 172.21.15.107:0/3460734364 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649243dbb50 con 0x5649244f7190 2024-04-24T23:46:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff2657fa640 10 client.15984 handle_command_reply: tid=0 2024-04-24T23:46:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 client.15984 shutdown 2024-04-24T23:46:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 msgr2=0x56492448ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff2780cc240 tx=0x7ff27800c760 comp rx=0 tx=0).stop 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 shutdown_connections 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5649244f7190 0x5649244f9580 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff2657fa640 0 client.15984 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564924489850 0x564924489cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56492447f100 0x564924489310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 --2- 172.21.15.107:0/3460734364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564924479e10 0x56492448ad00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 >> 172.21.15.107:0/3460734364 conn(0x564924438950 msgr2=0x564924478e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 shutdown_connections 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 1 -- 172.21.15.107:0/3460734364 wait complete. 2024-04-24T23:46:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.434+0000 7ff286b99740 20 client.15984 trim_cache size 0 max 16384 2024-04-24T23:46:27.441 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:46:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-04-24T23:46:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "5f8cd440-c604-4961-8246-05ae1f0787bf", 2024-04-24T23:46:27.441 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-04-24T23:46:27.441 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 msgr2=0x7ff2680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 0x7ff2680a5050 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x56492446ff70 tx=0x7ff274031bd0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 msgr2=0x7ff27c196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff278007b50 tx=0x7ff27800e420 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 shutdown_connections 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff2680a2b90 0x7ff2680a5050 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff27c102230 0x7ff27c19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff27c101870 0x7ff27c196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 --2- 172.21.15.107:0/1816792764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff27c100680 0x7ff27c196970 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 >> 172.21.15.107:0/1816792764 conn(0x7ff27c0fbbf0 msgr2=0x7ff27c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 shutdown_connections 2024-04-24T23:46:27.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.435+0000 7ff283da9640 1 -- 172.21.15.107:0/1816792764 wait complete. 2024-04-24T23:46:27.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:46:27.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/170515541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74107460 msgr2=0x7f4e741098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/170515541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74107460 0x7f4e741098b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009ca0 tx=0x7f4e6c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/170515541 shutdown_connections 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/170515541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74107460 0x7f4e741098b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/170515541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74104b30 0x7f4e74106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/170515541 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74102200 0x7f4e741045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/170515541 >> 172.21.15.107:0/170515541 conn(0x7f4e740fbbf0 msgr2=0x7f4e740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/170515541 shutdown_connections 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.548+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/170515541 wait complete. 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 Processor -- start 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 -- start start 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74104b30 0x7f4e741001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74107460 0x7f4e741006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410d420 con 0x7f4e74102200 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410d2a0 con 0x7f4e74107460 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7c6a3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410d5a0 con 0x7f4e74104b30 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7a43f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7a43f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59832/0 (socket says 172.21.15.107:59832) 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7a43f640 1 -- 172.21.15.107:0/1686536493 learned_addr learned my addr 172.21.15.107:0/1686536493 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.549+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74107460 0x7f4e741006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74104b30 0x7f4e741001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 -- 172.21.15.107:0/1686536493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74104b30 msgr2=0x7f4e741001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74104b30 0x7f4e741001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 -- 172.21.15.107:0/1686536493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74107460 msgr2=0x7f4e741006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74107460 0x7f4e741006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 -- 172.21.15.107:0/1686536493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e74100f80 con 0x7f4e74102200 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e6b7fe640 1 -- 172.21.15.107:0/1686536493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000cd80 con 0x7f4e74102200 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741a1bb0 con 0x7f4e74102200 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e6b7fe640 1 -- 172.21.15.107:0/1686536493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e70005500 con 0x7f4e74102200 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e6b7fe640 1 -- 172.21.15.107:0/1686536493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70010690 con 0x7f4e74102200 2024-04-24T23:46:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.550+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e741a20b0 con 0x7f4e74102200 2024-04-24T23:46:27.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.554+0000 7f4e7f3a8740 1 Processor -- start 2024-04-24T23:46:27.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.554+0000 7f4e6b7fe640 1 -- 172.21.15.107:0/1686536493 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4e700046c0 con 0x7f4e74102200 2024-04-24T23:46:27.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.554+0000 7f4e7f3a8740 1 -- start start 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.554+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.554+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e4725730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e4728230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e6b7fe640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 0x7f4e600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59834/0 (socket says 172.21.15.107:59834) 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 -- 172.21.15.107:0/2619912227 learned_addr learned my addr 172.21.15.107:0/2619912227 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640e46846b0 con 0x5640e4720240 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5640e462c880 con 0x5640e4725da0 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e6b7fe640 1 -- 172.21.15.107:0/1686536493 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e700cf940 con 0x7f4e74102200 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e4728230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e4725730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5640e4680670 con 0x5640e4720c30 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 0x7f4e600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 -- 172.21.15.107:0/2619912227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 msgr2=0x5640e4725730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e4725730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 -- 172.21.15.107:0/2619912227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 msgr2=0x5640e4728230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e4728230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.555+0000 7f4e7ac40640 1 -- 172.21.15.107:0/2619912227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e44627d0 con 0x5640e4720240 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.556+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c035000 tx=0x7f4e6c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b435db3ca6c41b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.556+0000 7f4e68ff9640 1 -- 172.21.15.107:0/2619912227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0318b0 con 0x5640e4720240 2024-04-24T23:46:27.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.556+0000 7f4e68ff9640 1 -- 172.21.15.107:0/2619912227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e6c04f070 con 0x5640e4720240 2024-04-24T23:46:27.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.556+0000 7f4e68ff9640 1 -- 172.21.15.107:0/2619912227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c02f880 con 0x5640e4720240 2024-04-24T23:46:27.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 msgr2=0x5640e4720660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c035000 tx=0x7f4e6c009a20 comp rx=0 tx=0).stop 2024-04-24T23:46:27.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 0x7f4e600a5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c04e000 tx=0x7f4e6c04c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 shutdown_connections 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e4728230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e4725730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2619912227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4720660 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 >> 172.21.15.107:0/2619912227 conn(0x5640e471b8b0 msgr2=0x5640e471bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 shutdown_connections 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2619912227 wait complete. 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 -- start start 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.557+0000 7f4e7f3a8740 1 Processor -- start 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4732c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e473a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640e4728bd0 con 0x5640e4720240 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5640e463ff50 con 0x5640e4720c30 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5640e4728d50 con 0x5640e4725da0 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7f3a8740 10 client.0 _ll_register_callbacks cb 0x5640e467b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:37630/0 (socket says 172.21.15.107:37630) 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 -- 172.21.15.107:0/2396172523 learned_addr learned my addr 172.21.15.107:0/2396172523 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4732c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 -- 172.21.15.107:0/2396172523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 msgr2=0x5640e473a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e473a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 -- 172.21.15.107:0/2396172523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 msgr2=0x5640e4732c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4732c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 -- 172.21.15.107:0/2396172523 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e43983f0 con 0x5640e4720c30 2024-04-24T23:46:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4732c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009d80 tx=0x7f4e640117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a41928884f08121 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.558+0000 7f4e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e64012860 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e6c04f070 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e64012a00 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e64012c20 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e7f3a8740 10 client.25397 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640e43580f0 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e6c04f070 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.559+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4e6400f040 con 0x5640e4720c30 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 10 client.25397 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 10 client.25397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:27.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 20 client.25397 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e473ebd0 0x5640e4740fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 4 client.25397 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640e4691bf0 con 0x5640e473ebd0 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e473ebd0 0x5640e4740fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e5dffb640 10 client.25397 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.560+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e473ebd0 0x5640e4740fe0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecfa853732996c1d server_cookie=5484d316666ae54c in_seq=0 out_seq=0 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e5dffb640 1 -- 172.21.15.107:0/2396172523 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640e4691bf0 con 0x5640e473ebd0 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e5dffb640 10 client.25397 handle_command_reply: tid=0 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 client.25397 shutdown 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 msgr2=0x5640e47331b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009d80 tx=0x7f4e640117b0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 shutdown_connections 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e473ebd0 0x5640e4740fe0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e473a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720c30 0x5640e47331b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e5dffb640 0 client.25397 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2396172523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e4732c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 >> 172.21.15.107:0/2396172523 conn(0x5640e471b8b0 msgr2=0x5640e471aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 shutdown_connections 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2396172523 wait complete. 2024-04-24T23:46:27.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.561+0000 7f4e7f3a8740 20 client.25397 trim_cache size 0 max 16384 2024-04-24T23:46:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 Processor -- start 2024-04-24T23:46:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 -- start start 2024-04-24T23:46:27.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e473a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e471e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a82f0 con 0x5640e4720240 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a9e30 con 0x5640e4725da0 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7f3a8740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a89c0 con 0x5640e4720c30 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59858/0 (socket says 172.21.15.107:59858) 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 -- 172.21.15.107:0/2023403837 learned_addr learned my addr 172.21.15.107:0/2023403837 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e473a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e471e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 -- 172.21.15.107:0/2023403837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 msgr2=0x5640e473a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e473a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 -- 172.21.15.107:0/2023403837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 msgr2=0x5640e471e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e471e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7a43f640 1 -- 172.21.15.107:0/2023403837 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e471b3f0 con 0x5640e4720240 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.579+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e471e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.580+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4e700987a0 tx=0x7f4e7009c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93195fd081930ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.580+0000 7f4e5ffff640 1 -- 172.21.15.107:0/2023403837 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e700b8070 con 0x5640e4720240 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.580+0000 7f4e5ffff640 1 -- 172.21.15.107:0/2023403837 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e700a4a40 con 0x5640e4720240 2024-04-24T23:46:27.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.580+0000 7f4e5ffff640 1 -- 172.21.15.107:0/2023403837 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7009f040 con 0x5640e4720240 2024-04-24T23:46:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2023403837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 msgr2=0x5640e471bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4e700987a0 tx=0x7f4e7009c4d0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2023403837 shutdown_connections 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4725da0 0x5640e471e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4720c30 0x5640e473a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/2023403837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720240 0x5640e471bc90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2023403837 >> 172.21.15.107:0/2023403837 conn(0x5640e471b8b0 msgr2=0x5640e4722000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2023403837 shutdown_connections 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/2023403837 wait complete. 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 -- start start 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.581+0000 7f4e7f3a8740 1 Processor -- start 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 0x5640e4737a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59860/0 (socket says 172.21.15.107:59860) 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 -- 172.21.15.107:0/3550691485 learned_addr learned my addr 172.21.15.107:0/3550691485 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e4738fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 0x5640e4737a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a9760 con 0x5640e4720c30 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a9090 con 0x5640e4720240 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741a62d0 con 0x5640e4725da0 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7f3a8740 10 client.0 _ll_register_callbacks cb 0x7f4e64009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7ac40640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e4738fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 -- 172.21.15.107:0/3550691485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 msgr2=0x5640e4738fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e4738fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 -- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 msgr2=0x5640e4737a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 0x5640e4737a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e4739590 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 0x5640e4737a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e79c3e640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x5640e4734150 tx=0x7f4e64009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77809b015a82630 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.582+0000 7f4e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e64032020 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e64005410 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e700a4a40 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e64005610 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640e47a52b0 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e700a4a40 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4e6402d070 con 0x5640e4720c30 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:27.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 20 client.16002 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e47a5d30 0x5640e47a8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 4 client.16002 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.583+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640e468bf40 con 0x5640e47a5d30 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e47a5d30 0x5640e47a8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e5dffb640 10 client.16002 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e7a43f640 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e47a5d30 0x5640e47a8150 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10948495cac76805 server_cookie=f8cbcc17345cd6a2 in_seq=0 out_seq=0 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e5dffb640 1 -- 172.21.15.107:0/3550691485 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5640e468bf40 con 0x5640e47a5d30 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e5dffb640 10 client.16002 handle_command_reply: tid=0 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.584+0000 7f4e7f3a8740 1 client.16002 shutdown 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 msgr2=0x5640e4737f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x5640e4734150 tx=0x7f4e64009f50 comp rx=0 tx=0).stop 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 shutdown_connections 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5640e47a5d30 0x5640e47a8150 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5640e4725da0 0x5640e4738fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e5dffb640 0 client.16002 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640e4720c30 0x5640e4737f90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 --2- 172.21.15.107:0/3550691485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5640e4720240 0x5640e4737a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 >> 172.21.15.107:0/3550691485 conn(0x5640e471b8b0 msgr2=0x5640e471a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 shutdown_connections 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 1 -- 172.21.15.107:0/3550691485 wait complete. 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7f3a8740 20 client.16002 trim_cache size 0 max 16384 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:46:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "5f8cd440-c604-4961-8246-05ae1f0787bf": { 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "5f8cd440-c604-4961-8246-05ae1f0787bf", 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:46:27.591 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:46:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 msgr2=0x7f4e600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.585+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 0x7f4e600a5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c04e000 tx=0x7f4e6c04c000 comp rx=0 tx=0).stop 2024-04-24T23:46:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 msgr2=0x7f4e74101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000edf0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 shutdown_connections 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4e600a2b90 0x7f4e600a5050 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e74107460 0x7f4e741006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4e74104b30 0x7f4e741001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 --2- 172.21.15.107:0/1686536493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e74102200 0x7f4e74101b70 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 >> 172.21.15.107:0/1686536493 conn(0x7f4e740fbbf0 msgr2=0x7f4e74102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 shutdown_connections 2024-04-24T23:46:27.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:27.586+0000 7f4e7c6a3640 1 -- 172.21.15.107:0/1686536493 wait complete. 2024-04-24T23:46:27.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:27 smithi107 ceph-mon[29118]: scrub queued for path: / 2024-04-24T23:46:27.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:27 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:27.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:27 smithi107 ceph-mon[29118]: scrub summary: active paths [/] 2024-04-24T23:46:27.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:27 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:27.598 INFO:tasks.cephfs.filesystem:scrub status for tag:5f8cd440-c604-4961-8246-05ae1f0787bf - {'path': '/', 'tag': '5f8cd440-c604-4961-8246-05ae1f0787bf', 'options': 'recursive,force'} 2024-04-24T23:46:28.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:27 smithi195 ceph-mon[33019]: scrub queued for path: / 2024-04-24T23:46:28.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:27 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:28.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:27 smithi195 ceph-mon[33019]: scrub summary: active paths [/] 2024-04-24T23:46:28.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:27 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:28.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:27 smithi170 ceph-mon[32714]: scrub queued for path: / 2024-04-24T23:46:28.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:27 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:28.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:27 smithi170 ceph-mon[32714]: scrub summary: active paths [/] 2024-04-24T23:46:28.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:27 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:46:28.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:28 smithi107 ceph-mon[29118]: pgmap v403: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:29.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:28 smithi195 ceph-mon[33019]: pgmap v403: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:29.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:28 smithi170 ceph-mon[32714]: pgmap v403: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:29.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:29 smithi107 ceph-mon[29118]: scrub summary: idle 2024-04-24T23:46:29.963 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:29 smithi195 ceph-mon[33019]: scrub summary: idle 2024-04-24T23:46:30.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:29 smithi170 ceph-mon[32714]: scrub summary: idle 2024-04-24T23:46:30.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:30 smithi107 ceph-mon[29118]: pgmap v404: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:31.062 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:30 smithi195 ceph-mon[33019]: pgmap v404: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:31.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:30 smithi170 ceph-mon[32714]: pgmap v404: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T23:46:31.498 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-04-24T23:46:31.499 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-24T23:46:31.502 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T23:46:31.532 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T23:46:31.534 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T23:46:31.534 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T23:46:32.173 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T23:46:32.228 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T23:46:32.269 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T23:46:32.270 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T23:46:32.270 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T23:46:32.308 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T23:46:32.311 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:46:32.311 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T23:46:32.367 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-04-24T23:46:32.368 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-04-24T23:46:32.368 DEBUG:teuthology.orchestra.run.smithi107:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-04-24T23:46:32.427 INFO:tasks.workunit.client.0.smithi107.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Iozone: Performance Test of File I/O 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Version $Revision: 3.494 $ 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Compiled for 64 bit mode. 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Build: linux-AMD64 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-24T23:46:32.504 INFO:tasks.workunit.client.0.smithi107.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Run began: Wed Apr 24 23:46:32 2024 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Include close in write timing 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Include fsync in write timing 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: File size set to 1048576 kB 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Record Size 16 kB 2024-04-24T23:46:32.505 INFO:tasks.workunit.client.0.smithi107.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Output is in kBytes/sec 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Time Resolution = 0.000001 seconds. 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache size set to 1024 kBytes. 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache line size set to 32 bytes. 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: File stride size set to 17 * record size. 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Throughput test with 1 process 2024-04-24T23:46:32.506 INFO:tasks.workunit.client.0.smithi107.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-04-24T23:46:33.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:33 smithi107 ceph-mon[29118]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-24T23:46:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:33 smithi195 ceph-mon[33019]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-24T23:46:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:33 smithi170 ceph-mon[32714]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-04-24T23:46:33.878 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 initial writers = 764024.31 kB/sec 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 initial writers = 763936.91 kB/sec 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 764024.31 kB/sec 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 764024.31 kB/sec 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 764024.31 kB/sec 2024-04-24T23:46:33.879 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:35.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:35 smithi107 ceph-mon[29118]: pgmap v406: 129 pgs: 129 active+clean; 105 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-04-24T23:46:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:35 smithi195 ceph-mon[33019]: pgmap v406: 129 pgs: 129 active+clean; 105 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-04-24T23:46:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:35 smithi170 ceph-mon[32714]: pgmap v406: 129 pgs: 129 active+clean; 105 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-04-24T23:46:37.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:37 smithi107 ceph-mon[29118]: pgmap v407: 129 pgs: 129 active+clean; 653 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 54 MiB/s wr, 26 op/s 2024-04-24T23:46:37.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:37 smithi170 ceph-mon[32714]: pgmap v407: 129 pgs: 129 active+clean; 653 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 54 MiB/s wr, 26 op/s 2024-04-24T23:46:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:37 smithi195 ceph-mon[33019]: pgmap v407: 129 pgs: 129 active+clean; 653 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 54 MiB/s wr, 26 op/s 2024-04-24T23:46:39.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:39 smithi107 ceph-mon[29118]: pgmap v408: 129 pgs: 129 active+clean; 969 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 81 MiB/s wr, 31 op/s 2024-04-24T23:46:39.472 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:39.472 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 rewriters = 817682.12 kB/sec 2024-04-24T23:46:39.472 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 rewriters = 817561.79 kB/sec 2024-04-24T23:46:39.472 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 817682.12 kB/sec 2024-04-24T23:46:39.472 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 817682.12 kB/sec 2024-04-24T23:46:39.473 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 817682.12 kB/sec 2024-04-24T23:46:39.473 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:39.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:39 smithi195 ceph-mon[33019]: pgmap v408: 129 pgs: 129 active+clean; 969 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 81 MiB/s wr, 31 op/s 2024-04-24T23:46:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:39 smithi170 ceph-mon[32714]: pgmap v408: 129 pgs: 129 active+clean; 969 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 81 MiB/s wr, 31 op/s 2024-04-24T23:46:41.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:41 smithi107 ceph-mon[29118]: pgmap v409: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 119 MiB/s wr, 41 op/s 2024-04-24T23:46:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:41 smithi195 ceph-mon[33019]: pgmap v409: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 119 MiB/s wr, 41 op/s 2024-04-24T23:46:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:41 smithi170 ceph-mon[32714]: pgmap v409: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 119 MiB/s wr, 41 op/s 2024-04-24T23:46:42.225 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 readers = 6858682.50 kB/sec 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 readers = 6849762.94 kB/sec 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 6858682.50 kB/sec 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 6858682.50 kB/sec 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 6858682.50 kB/sec 2024-04-24T23:46:42.226 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:43.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:43 smithi107 ceph-mon[29118]: pgmap v410: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 130 MiB/s wr, 45 op/s 2024-04-24T23:46:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:43 smithi195 ceph-mon[33019]: pgmap v410: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 130 MiB/s wr, 45 op/s 2024-04-24T23:46:43.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:43 smithi170 ceph-mon[32714]: pgmap v410: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 130 MiB/s wr, 45 op/s 2024-04-24T23:46:44.448 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:44.448 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 re-readers = 6950424.50 kB/sec 2024-04-24T23:46:44.448 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 re-readers = 6940530.90 kB/sec 2024-04-24T23:46:44.448 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 6950424.50 kB/sec 2024-04-24T23:46:44.449 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 6950424.50 kB/sec 2024-04-24T23:46:44.449 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 6950424.50 kB/sec 2024-04-24T23:46:44.449 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:45.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:45 smithi107 ceph-mon[29118]: pgmap v411: 129 pgs: 129 active+clean; 1.0 GiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 158 MiB/s wr, 50 op/s 2024-04-24T23:46:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:45 smithi195 ceph-mon[33019]: pgmap v411: 129 pgs: 129 active+clean; 1.0 GiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 158 MiB/s wr, 50 op/s 2024-04-24T23:46:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:45 smithi170 ceph-mon[32714]: pgmap v411: 129 pgs: 129 active+clean; 1.0 GiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 158 MiB/s wr, 50 op/s 2024-04-24T23:46:47.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:47 smithi107 ceph-mon[29118]: pgmap v412: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 162 MiB/s wr, 54 op/s 2024-04-24T23:46:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:47 smithi195 ceph-mon[33019]: pgmap v412: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 162 MiB/s wr, 54 op/s 2024-04-24T23:46:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:47 smithi170 ceph-mon[32714]: pgmap v412: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 162 MiB/s wr, 54 op/s 2024-04-24T23:46:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:49 smithi107 ceph-mon[29118]: pgmap v413: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 116 MiB/s wr, 38 op/s 2024-04-24T23:46:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:49 smithi195 ceph-mon[33019]: pgmap v413: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 116 MiB/s wr, 38 op/s 2024-04-24T23:46:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:49 smithi170 ceph-mon[32714]: pgmap v413: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 116 MiB/s wr, 38 op/s 2024-04-24T23:46:50.754 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.755 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.755 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.755 INFO:tasks.workunit.client.0.smithi107.stdout:iozone test complete. 2024-04-24T23:46:50.755 INFO:tasks.workunit.client.0.smithi107.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: Iozone: Performance Test of File I/O 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: Version $Revision: 3.494 $ 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: Compiled for 64 bit mode. 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: Build: linux-AMD64 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.820 INFO:tasks.workunit.client.0.smithi107.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Run began: Wed Apr 24 23:46:50 2024 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Include close in write timing 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Include fsync in write timing 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: File size set to 1048576 kB 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Record Size 1024 kB 2024-04-24T23:46:50.821 INFO:tasks.workunit.client.0.smithi107.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Output is in kBytes/sec 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Time Resolution = 0.000001 seconds. 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache size set to 1024 kBytes. 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache line size set to 32 bytes. 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: File stride size set to 17 * record size. 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Throughput test with 1 process 2024-04-24T23:46:50.822 INFO:tasks.workunit.client.0.smithi107.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-04-24T23:46:51.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:51 smithi107 ceph-mon[29118]: pgmap v414: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 90 MiB/s wr, 32 op/s 2024-04-24T23:46:51.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:51 smithi195 ceph-mon[33019]: pgmap v414: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 90 MiB/s wr, 32 op/s 2024-04-24T23:46:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:51 smithi170 ceph-mon[32714]: pgmap v414: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 90 MiB/s wr, 32 op/s 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 initial writers = 807636.88 kB/sec 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 initial writers = 807500.66 kB/sec 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 807636.88 kB/sec 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 807636.88 kB/sec 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 807636.88 kB/sec 2024-04-24T23:46:52.150 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:53.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:53 smithi107 ceph-mon[29118]: pgmap v415: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 53 MiB/s wr, 24 op/s 2024-04-24T23:46:53.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:53 smithi195 ceph-mon[33019]: pgmap v415: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 53 MiB/s wr, 24 op/s 2024-04-24T23:46:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:53 smithi170 ceph-mon[32714]: pgmap v415: 129 pgs: 129 active+clean; 1.0 GiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 53 MiB/s wr, 24 op/s 2024-04-24T23:46:55.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:55 smithi107 ceph-mon[29118]: pgmap v416: 129 pgs: 129 active+clean; 1.2 GiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 68 MiB/s wr, 28 op/s 2024-04-24T23:46:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:55 smithi195 ceph-mon[33019]: pgmap v416: 129 pgs: 129 active+clean; 1.2 GiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 68 MiB/s wr, 28 op/s 2024-04-24T23:46:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:55 smithi170 ceph-mon[32714]: pgmap v416: 129 pgs: 129 active+clean; 1.2 GiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 68 MiB/s wr, 28 op/s 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 rewriters = 843163.56 kB/sec 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 rewriters = 843051.10 kB/sec 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 843163.56 kB/sec 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 843163.56 kB/sec 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 843163.56 kB/sec 2024-04-24T23:46:55.736 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:57 smithi195 ceph-mon[33019]: pgmap v417: 129 pgs: 129 active+clean; 1.2 GiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 49 op/s 2024-04-24T23:46:57.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:57 smithi170 ceph-mon[32714]: pgmap v417: 129 pgs: 129 active+clean; 1.2 GiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 49 op/s 2024-04-24T23:46:57.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:46:57.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:46:57.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:57 smithi107 ceph-mon[29118]: pgmap v417: 129 pgs: 129 active+clean; 1.2 GiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 49 op/s 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 -- 172.21.15.107:0/1099471889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80107460 msgr2=0x7f8c801098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 --2- 172.21.15.107:0/1099471889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80107460 0x7f8c801098b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ca0 tx=0x7f8c7802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 -- 172.21.15.107:0/1099471889 shutdown_connections 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 --2- 172.21.15.107:0/1099471889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80107460 0x7f8c801098b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 --2- 172.21.15.107:0/1099471889 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80104b30 0x7f8c80106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 --2- 172.21.15.107:0/1099471889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80102200 0x7f8c801045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.713+0000 7f8c8861c640 1 -- 172.21.15.107:0/1099471889 >> 172.21.15.107:0/1099471889 conn(0x7f8c800fbbf0 msgr2=0x7f8c800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- 172.21.15.107:0/1099471889 shutdown_connections 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- 172.21.15.107:0/1099471889 wait complete. 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 Processor -- start 2024-04-24T23:46:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- start start 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80102200 0x7f8c80197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 0x7f8c8019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010d470 con 0x7f8c80104b30 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010d2f0 con 0x7f8c80107460 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c8861c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010d5f0 con 0x7f8c80102200 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c85bb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c85bb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44452/0 (socket says 172.21.15.107:44452) 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.714+0000 7f8c85bb7640 1 -- 172.21.15.107:0/2827307080 learned_addr learned my addr 172.21.15.107:0/2827307080 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c863b8640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80102200 0x7f8c80197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 0x7f8c8019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 -- 172.21.15.107:0/2827307080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80102200 msgr2=0x7f8c80197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80102200 0x7f8c80197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 -- 172.21.15.107:0/2827307080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 msgr2=0x7f8c8019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 0x7f8c8019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 -- 172.21.15.107:0/2827307080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8019f700 con 0x7f8c80104b30 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 0x7f8c8019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000b450 tx=0x7f8c7000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c777fe640 1 -- 172.21.15.107:0/2827307080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70015020 con 0x7f8c80104b30 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c777fe640 1 -- 172.21.15.107:0/2827307080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c70011d70 con 0x7f8c80104b30 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8019f9f0 con 0x7f8c80104b30 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c777fe640 1 -- 172.21.15.107:0/2827307080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000f340 con 0x7f8c80104b30 2024-04-24T23:46:57.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.715+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8019ff70 con 0x7f8c80104b30 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.718+0000 7f8c777fe640 1 -- 172.21.15.107:0/2827307080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8c70018020 con 0x7f8c80104b30 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.718+0000 7f8c8b408740 1 Processor -- start 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.718+0000 7f8c8b408740 1 -- start start 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c15f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c167230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55855c0c36b0 con 0x55855c15fc30 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55855c06b880 con 0x55855c164da0 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c8b408740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55855c0bf670 con 0x55855c15f240 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44464/0 (socket says 172.21.15.107:44464) 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 -- 172.21.15.107:0/475992382 learned_addr learned my addr 172.21.15.107:0/475992382 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c167230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c777fe640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 0x7f8c6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 -- 172.21.15.107:0/475992382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 msgr2=0x55855c15f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c15f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 -- 172.21.15.107:0/475992382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 msgr2=0x55855c167230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c167230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 -- 172.21.15.107:0/475992382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855bea17d0 con 0x55855c15fc30 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c167230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 0x7f8c6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c863b8640 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00cbf0 tx=0x7f8c7c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf10e94f8284ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c777fe640 1 -- 172.21.15.107:0/2827307080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c700cf000 con 0x7f8c80104b30 2024-04-24T23:46:57.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c74bfe640 1 -- 172.21.15.107:0/475992382 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c010630 con 0x55855c15fc30 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.719+0000 7f8c74bfe640 1 -- 172.21.15.107:0/475992382 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c010cb0 con 0x55855c15fc30 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c74bfe640 1 -- 172.21.15.107:0/475992382 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c019c30 con 0x55855c15fc30 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 -- 172.21.15.107:0/475992382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 msgr2=0x55855c164730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 0x7f8c6c0a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009430 tx=0x7f8c780095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00cbf0 tx=0x7f8c7c00e720 comp rx=0 tx=0).stop 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 -- 172.21.15.107:0/475992382 shutdown_connections 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c167230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15fc30 0x55855c164730 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 --2- 172.21.15.107:0/475992382 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c15f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 -- 172.21.15.107:0/475992382 >> 172.21.15.107:0/475992382 conn(0x55855c15a8b0 msgr2=0x55855c15acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 -- 172.21.15.107:0/475992382 shutdown_connections 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.720+0000 7f8c8b408740 1 -- 172.21.15.107:0/475992382 wait complete. 2024-04-24T23:46:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 -- start start 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 Processor -- start 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c169660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c169ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55855c167d50 con 0x55855c164da0 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55855c07ef50 con 0x55855c15fc30 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55855c167ed0 con 0x55855c15f240 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c8b408740 10 client.0 _ll_register_callbacks cb 0x55855c0ba000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44470/0 (socket says 172.21.15.107:44470) 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 -- 172.21.15.107:0/3500096419 learned_addr learned my addr 172.21.15.107:0/3500096419 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c863b8640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c169660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c169ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 -- 172.21.15.107:0/3500096419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 msgr2=0x55855c169660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c169660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 -- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 msgr2=0x55855c169ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c169ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.721+0000 7f8c86bb9640 1 -- 172.21.15.107:0/3500096419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855bdd73f0 con 0x55855c164da0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ca0 tx=0x7f8c7802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1316c04228f8808 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c78004450 con 0x55855c164da0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c780045f0 con 0x55855c164da0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c7c010cb0 con 0x55855c164da0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c780047f0 con 0x55855c164da0 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55855bd970f0 con 0x55855c164da0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c010cb0 con 0x55855c164da0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8c78030770 con 0x55855c164da0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 20 client.16014 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c175500 0x55855c177940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 4 client.16014 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.722+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55855c0d0bf0 con 0x55855c175500 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.723+0000 7f8c863b8640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c175500 0x55855c177940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.723+0000 7f8c69ffb640 10 client.16014 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.723+0000 7f8c863b8640 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c175500 0x55855c177940 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=539afc9c7a127718 server_cookie=6d71a9a1ccaaa756 in_seq=0 out_seq=0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c69ffb640 1 -- 172.21.15.107:0/3500096419 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55855c0d0bf0 con 0x55855c175500 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c69ffb640 10 client.16014 handle_command_reply: tid=0 2024-04-24T23:46:57.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 client.16014 shutdown 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 msgr2=0x55855c170be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ca0 tx=0x7f8c7802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 shutdown_connections 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c175500 0x55855c177940 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c164da0 0x55855c170be0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c169ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 --2- 172.21.15.107:0/3500096419 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15f240 0x55855c169660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 >> 172.21.15.107:0/3500096419 conn(0x55855c15a8b0 msgr2=0x55855c159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c69ffb640 0 client.16014 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 shutdown_connections 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 1 -- 172.21.15.107:0/3500096419 wait complete. 2024-04-24T23:46:57.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.724+0000 7f8c8b408740 20 client.16014 trim_cache size 0 max 16384 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 Processor -- start 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 -- start start 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15fc30 0x55855c170ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c15d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801aa600 con 0x55855c15f240 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801ac140 con 0x55855c164da0 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c8b408740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801aacd0 con 0x55855c15fc30 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44476/0 (socket says 172.21.15.107:44476) 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 -- 172.21.15.107:0/745895032 learned_addr learned my addr 172.21.15.107:0/745895032 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15fc30 0x55855c170ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 -- 172.21.15.107:0/745895032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15fc30 msgr2=0x55855c170ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15fc30 0x55855c170ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 -- 172.21.15.107:0/745895032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 msgr2=0x55855c15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 -- 172.21.15.107:0/745895032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855c14c090 con 0x55855c15f240 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c863b8640 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00ec20 tx=0x7f8c7c005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a039b63c58f212e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c6bfff640 1 -- 172.21.15.107:0/745895032 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c010e60 con 0x55855c15f240 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c6bfff640 1 -- 172.21.15.107:0/745895032 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c005520 con 0x55855c15f240 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.742+0000 7f8c6bfff640 1 -- 172.21.15.107:0/745895032 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0048c0 con 0x55855c15f240 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- 172.21.15.107:0/745895032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 msgr2=0x55855c177510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00ec20 tx=0x7f8c7c005ce0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- 172.21.15.107:0/745895032 shutdown_connections 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c164da0 0x55855c15d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c15fc30 0x55855c170ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- 172.21.15.107:0/745895032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c15f240 0x55855c177510 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- 172.21.15.107:0/745895032 >> 172.21.15.107:0/745895032 conn(0x55855c15a8b0 msgr2=0x55855c160ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- 172.21.15.107:0/745895032 shutdown_connections 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- 172.21.15.107:0/745895032 wait complete. 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- start start 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 Processor -- start 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c16e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 0x55855c16eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801aba70 con 0x55855c16fa60 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801ab3a0 con 0x55855c15fc30 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8c801a85e0 con 0x55855c164da0 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.743+0000 7f8c8b408740 10 client.0 _ll_register_callbacks cb 0x7f8c78009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:57.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44478/0 (socket says 172.21.15.107:44478) 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 -- 172.21.15.107:0/477663288 learned_addr learned my addr 172.21.15.107:0/477663288 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c863b8640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c16e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 -- 172.21.15.107:0/477663288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 msgr2=0x55855c16eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 0x55855c16eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 0x55855c16eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 -- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 msgr2=0x55855c16e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c16e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 -- 172.21.15.107:0/477663288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855c170420 con 0x55855c16fa60 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c863b8640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c16e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c85bb7640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 0x55855c16eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c86bb9640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8c7802a900 tx=0x7f8c78004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af6187d99aca044 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780307c0 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c78030960 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c7c005520 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c78030e50 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c8b408740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55855c1dc0a0 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.744+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c005520 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8c78051730 con 0x55855c16fa60 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 20 client.16020 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c1dc8e0 0x55855c1ded00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 4 client.16020 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:46:57.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55855c0caf40 con 0x55855c1dc8e0 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c863b8640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c1dc8e0 0x55855c1ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c69ffb640 10 client.16020 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c863b8640 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c1dc8e0 0x55855c1ded00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=582fafcde4c09c9a server_cookie=ea9d1df037c03a2a in_seq=0 out_seq=0 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c69ffb640 1 -- 172.21.15.107:0/477663288 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55855c0caf40 con 0x55855c1dc8e0 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c69ffb640 10 client.16020 handle_command_reply: tid=0 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.745+0000 7f8c8b408740 1 client.16020 shutdown 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 msgr2=0x55855c16fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8c7802a900 tx=0x7f8c78004770 comp rx=0 tx=0).stop 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 shutdown_connections 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55855c1dc8e0 0x55855c1ded00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55855c16fa60 0x55855c16fee0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55855c164da0 0x55855c16eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c69ffb640 0 client.16020 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 --2- 172.21.15.107:0/477663288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55855c15fc30 0x55855c16e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 >> 172.21.15.107:0/477663288 conn(0x55855c15a8b0 msgr2=0x55855c159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 shutdown_connections 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 1 -- 172.21.15.107:0/477663288 wait complete. 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8b408740 20 client.16020 trim_cache size 0 max 16384 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-04-24T23:46:57.753 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 msgr2=0x7f8c6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 0x7f8c6c0a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009430 tx=0x7f8c780095c0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 msgr2=0x7f8c80198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000b450 tx=0x7f8c7000b910 comp rx=0 tx=0).stop 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 shutdown_connections 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f8c6c0a2b90 0x7f8c6c0a5050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c80107460 0x7f8c8019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c80104b30 0x7f8c80198020 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 --2- 172.21.15.107:0/2827307080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8c80102200 0x7f8c80197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 >> 172.21.15.107:0/2827307080 conn(0x7f8c800fbbf0 msgr2=0x7f8c80102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 shutdown_connections 2024-04-24T23:46:57.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.746+0000 7f8c8861c640 1 -- 172.21.15.107:0/2827307080 wait complete. 2024-04-24T23:46:57.757 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T23:46:57.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:46:57.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 -- 172.21.15.107:0/3706828524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff360101890 msgr2=0x7ff360101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 --2- 172.21.15.107:0/3706828524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff360101890 0x7ff360101d10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35402a410 comp rx=0 tx=0).stop 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 -- 172.21.15.107:0/3706828524 shutdown_connections 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 --2- 172.21.15.107:0/3706828524 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff360108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 --2- 172.21.15.107:0/3706828524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff360101890 0x7ff360101d10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 --2- 172.21.15.107:0/3706828524 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff3601006a0 0x7ff360100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 -- 172.21.15.107:0/3706828524 >> 172.21.15.107:0/3706828524 conn(0x7ff3600fbc30 msgr2=0x7ff3600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 -- 172.21.15.107:0/3706828524 shutdown_connections 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.895+0000 7ff365a72640 1 -- 172.21.15.107:0/3706828524 wait complete. 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 Processor -- start 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 -- start start 2024-04-24T23:46:57.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff360101890 0x7ff36019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff36019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff36010c300 con 0x7ff3601006a0 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff36010c180 con 0x7ff360102250 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff365a72640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff36010c480 con 0x7ff360101890 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44500/0 (socket says 172.21.15.107:44500) 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 -- 172.21.15.107:0/1905034720 learned_addr learned my addr 172.21.15.107:0/1905034720 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35effd640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff360101890 0x7ff36019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35ffff640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff36019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 -- 172.21.15.107:0/1905034720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff360101890 msgr2=0x7ff36019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff360101890 0x7ff36019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 -- 172.21.15.107:0/1905034720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 msgr2=0x7ff36019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff36019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:57.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35f7fe640 1 -- 172.21.15.107:0/1905034720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36019f640 con 0x7ff3601006a0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.896+0000 7ff35ffff640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff36019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff35f7fe640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff35000c970 tx=0x7ff35000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350007c10 con 0x7ff3601006a0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36019f930 con 0x7ff3601006a0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff350007db0 con 0x7ff3601006a0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35000f8a0 con 0x7ff3601006a0 2024-04-24T23:46:57.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.897+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36019fea0 con 0x7ff3601006a0 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.898+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3601079c0 con 0x7ff3601006a0 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.899+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff35000fb40 con 0x7ff3601006a0 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.899+0000 7ff35cff9640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 0x7ff3480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.900+0000 7ff35effd640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 0x7ff3480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.900+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff3500cf7e0 con 0x7ff3601006a0 2024-04-24T23:46:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.900+0000 7ff35effd640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 0x7ff3480a5050 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff3540314e0 tx=0x7ff354000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:57.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:57.904+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff35009c3a0 con 0x7ff3601006a0 2024-04-24T23:46:58.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.119+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff36002d170 con 0x7ff3601006a0 2024-04-24T23:46:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.119+0000 7ff35cff9640 1 -- 172.21.15.107:0/1905034720 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7ff36002d170 con 0x7ff3601006a0 2024-04-24T23:46:58.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:46:58.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:46:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:46:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 msgr2=0x7ff3480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 0x7ff3480a5050 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff3540314e0 tx=0x7ff354000bc0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 msgr2=0x7ff36006a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff35000c970 tx=0x7ff35000ce30 comp rx=0 tx=0).stop 2024-04-24T23:46:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 shutdown_connections 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff3480a2b90 0x7ff3480a5050 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff360102250 0x7ff36019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff360101890 0x7ff36019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 --2- 172.21.15.107:0/1905034720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3601006a0 0x7ff36006a930 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.121+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 >> 172.21.15.107:0/1905034720 conn(0x7ff3600fbc30 msgr2=0x7ff360104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.122+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 shutdown_connections 2024-04-24T23:46:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.122+0000 7ff365a72640 1 -- 172.21.15.107:0/1905034720 wait complete. 2024-04-24T23:46:58.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- 172.21.15.107:0/4007390953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 msgr2=0x7ff78c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4007390953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c104650 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009900 tx=0x7ff77c02a410 comp rx=0 tx=0).stop 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- 172.21.15.107:0/4007390953 shutdown_connections 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4007390953 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 0x7ff78c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4007390953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 0x7ff78c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4007390953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c104650 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- 172.21.15.107:0/4007390953 >> 172.21.15.107:0/4007390953 conn(0x7ff78c0fbbf0 msgr2=0x7ff78c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- 172.21.15.107:0/4007390953 shutdown_connections 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- 172.21.15.107:0/4007390953 wait complete. 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 Processor -- start 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.233+0000 7ff791ad4640 1 -- start start 2024-04-24T23:46:58.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 0x7ff78c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 0x7ff78c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c10d420 con 0x7ff78c102260 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c10d2a0 con 0x7ff78c104b90 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c10d5a0 con 0x7ff78c1074c0 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44520/0 (socket says 172.21.15.107:44520) 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 -- 172.21.15.107:0/4013531892 learned_addr learned my addr 172.21.15.107:0/4013531892 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff783fff640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 0x7ff78c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78bfff640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 0x7ff78c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 -- 172.21.15.107:0/4013531892 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 msgr2=0x7ff78c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 0x7ff78c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 -- 172.21.15.107:0/4013531892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 msgr2=0x7ff78c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 0x7ff78c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 -- 172.21.15.107:0/4013531892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c19f720 con 0x7ff78c102260 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009230 tx=0x7ff77c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff7897fa640 1 -- 172.21.15.107:0/4013531892 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c03f030 con 0x7ff78c102260 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c19f9b0 con 0x7ff78c102260 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff7897fa640 1 -- 172.21.15.107:0/4013531892 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff77c003ce0 con 0x7ff78c102260 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff7897fa640 1 -- 172.21.15.107:0/4013531892 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c035070 con 0x7ff78c102260 2024-04-24T23:46:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.234+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c19fef0 con 0x7ff78c102260 2024-04-24T23:46:58.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7897fa640 1 -- 172.21.15.107:0/4013531892 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff77c042030 con 0x7ff78c102260 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 Processor -- start 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 -- start start 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98ec20 0x55da2f993720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f996220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55da2f8f2670 con 0x55da2f98e230 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55da2f89a970 con 0x55da2f993d90 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7947e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55da2f8ee630 con 0x55da2f98ec20 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff78bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff7897fa640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 0x7ff7780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff78b7fe640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98ec20 0x55da2f993720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff783fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f996220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff78bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44536/0 (socket says 172.21.15.107:44536) 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff78bfff640 1 -- 172.21.15.107:0/3951137577 learned_addr learned my addr 172.21.15.107:0/3951137577 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.237+0000 7ff78bfff640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 0x7ff7780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff7897fa640 1 -- 172.21.15.107:0/4013531892 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff77c0efd50 con 0x7ff78c102260 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 -- 172.21.15.107:0/3951137577 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98ec20 msgr2=0x55da2f993720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98ec20 0x55da2f993720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 -- 172.21.15.107:0/3951137577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 msgr2=0x55da2f996220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f996220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 -- 172.21.15.107:0/3951137577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2f8be400 con 0x55da2f98e230 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff783fff640 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f996220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff784013de0 tx=0x7ff7840117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf78237f331bd48 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff78bfff640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 0x7ff7780a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff784011cb0 tx=0x7ff78401d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff781ffb640 1 -- 172.21.15.107:0/3951137577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784022070 con 0x55da2f98e230 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff781ffb640 1 -- 172.21.15.107:0/3951137577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff784003e50 con 0x55da2f98e230 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.238+0000 7ff781ffb640 1 -- 172.21.15.107:0/3951137577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784004630 con 0x55da2f98e230 2024-04-24T23:46:58.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- 172.21.15.107:0/3951137577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 msgr2=0x55da2f98e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff784013de0 tx=0x7ff7840117b0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- 172.21.15.107:0/3951137577 shutdown_connections 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f996220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98ec20 0x55da2f993720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- 172.21.15.107:0/3951137577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2f98e650 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- 172.21.15.107:0/3951137577 >> 172.21.15.107:0/3951137577 conn(0x55da2f9898a0 msgr2=0x55da2f989ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- 172.21.15.107:0/3951137577 shutdown_connections 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- 172.21.15.107:0/3951137577 wait complete. 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- start start 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 Processor -- start 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f998390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f99f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55da2f996ce0 con 0x55da2f98ec20 2024-04-24T23:46:58.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55da2f8ae050 con 0x55da2f993d90 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.239+0000 7ff7947e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55da2f996e60 con 0x55da2f98e230 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff7947e3740 10 client.0 _ll_register_callbacks cb 0x55da2f605be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44546/0 (socket says 172.21.15.107:44546) 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 -- 172.21.15.107:0/1324461943 learned_addr learned my addr 172.21.15.107:0/1324461943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f998390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff78bfff640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f99f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 -- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 msgr2=0x55da2f998390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f998390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 -- 172.21.15.107:0/1324461943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 msgr2=0x55da2f99f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f99f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 -- 172.21.15.107:0/1324461943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2f87fdc0 con 0x55da2f98ec20 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff783fff640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff77400bf50 tx=0x7ff7740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22513e657e925fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff76affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774004020 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff7740041c0 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff784003e50 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff7740043c0 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff7947e3740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.240+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da2f6063f0 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff784003e50 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff774010700 con 0x55da2f98ec20 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 10 client.16038 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 20 client.16038 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2f9a42e0 0x55da2f9a6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 4 client.16038 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da2f8ffbb0 con 0x55da2f9a42e0 2024-04-24T23:46:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2f9a42e0 0x55da2f9a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff76affd640 10 client.16038 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.241+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2f9a42e0 0x55da2f9a6720 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=561aa9668865ca0c server_cookie=6eeb5921fea5e3d9 in_seq=0 out_seq=0 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff76affd640 1 -- 172.21.15.107:0/1324461943 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da2f8ffbb0 con 0x55da2f9a42e0 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff76affd640 10 client.16038 handle_command_reply: tid=0 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 client.16038 shutdown 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 msgr2=0x55da2f9988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff77400bf50 tx=0x7ff7740118d0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 shutdown_connections 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2f9a42e0 0x55da2f9a6720 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f99f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f9988d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff76affd640 0 client.16038 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 --2- 172.21.15.107:0/1324461943 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f998390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 >> 172.21.15.107:0/1324461943 conn(0x55da2f9898a0 msgr2=0x55da2f988b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 shutdown_connections 2024-04-24T23:46:58.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 1 -- 172.21.15.107:0/1324461943 wait complete. 2024-04-24T23:46:58.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.242+0000 7ff7947e3740 20 client.16038 trim_cache size 0 max 16384 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.255+0000 7ff7947e3740 1 Processor -- start 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 -- start start 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f9979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f98fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1aa520 con 0x55da2f98ec20 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1ac060 con 0x55da2f993d90 2024-04-24T23:46:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7947e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1aabf0 con 0x55da2f98e230 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44554/0 (socket says 172.21.15.107:44554) 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 -- 172.21.15.107:0/853728094 learned_addr learned my addr 172.21.15.107:0/853728094 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff78bfff640 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f98fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f9979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 -- 172.21.15.107:0/853728094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 msgr2=0x55da2f9979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f9979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 -- 172.21.15.107:0/853728094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 msgr2=0x55da2f98fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f98fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 -- 172.21.15.107:0/853728094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2f6d07d0 con 0x55da2f98ec20 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff783fff640 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff774006010 tx=0x7ff77400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e99e0743684961 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7817fa640 1 -- 172.21.15.107:0/853728094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77402c070 con 0x55da2f98ec20 2024-04-24T23:46:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7817fa640 1 -- 172.21.15.107:0/853728094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff774010700 con 0x55da2f98ec20 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.256+0000 7ff7817fa640 1 -- 172.21.15.107:0/853728094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7740052e0 con 0x55da2f98ec20 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- 172.21.15.107:0/853728094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 msgr2=0x55da2f98f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff774006010 tx=0x7ff77400b8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- 172.21.15.107:0/853728094 shutdown_connections 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f993d90 0x55da2f98fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98ec20 0x55da2f98f670 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 --2- 172.21.15.107:0/853728094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f98e230 0x55da2f9979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- 172.21.15.107:0/853728094 >> 172.21.15.107:0/853728094 conn(0x55da2f9898a0 msgr2=0x55da2f988b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- 172.21.15.107:0/853728094 shutdown_connections 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- 172.21.15.107:0/853728094 wait complete. 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 -- start start 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.257+0000 7ff7947e3740 1 Processor -- start 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 0x55da2fa06b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f993d90 0x55da2fa0dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1ab990 con 0x55da2f98e230 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1ab2c0 con 0x55da2f98ec20 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1a8500 con 0x55da2f993d90 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 10 client.0 _ll_register_callbacks cb 0x7ff77400bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44558/0 (socket says 172.21.15.107:44558) 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 -- 172.21.15.107:0/573390273 learned_addr learned my addr 172.21.15.107:0/573390273 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78bfff640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f993d90 0x55da2fa0dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff783fff640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 0x55da2fa06b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 -- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f993d90 msgr2=0x55da2fa0dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f993d90 0x55da2fa0dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 -- 172.21.15.107:0/573390273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 msgr2=0x55da2fa06b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 0x55da2fa06b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 -- 172.21.15.107:0/573390273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2fa0e190 con 0x55da2f98e230 2024-04-24T23:46:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff783fff640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 0x55da2fa06b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff78b7fe640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009a20 tx=0x7ff77c0b8660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b264c819a389a80c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff76affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c049040 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff77c0ba590 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff774010700 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff77c0c0590 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 10 client.16044 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.258+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da2fa0f500 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff774010700 con 0x55da2f98e230 2024-04-24T23:46:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff77c0dc0b0 con 0x55da2f98e230 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 10 client.16044 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 10 client.16044 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 20 client.16044 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2fa0fe80 0x55da2fa122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 4 client.16044 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da2f998180 con 0x55da2fa0fe80 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff783fff640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2fa0fe80 0x55da2fa122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff76affd640 10 client.16044 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.259+0000 7ff783fff640 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2fa0fe80 0x55da2fa122a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2f9f23e1335d362f server_cookie=624a430b69c7af3e in_seq=0 out_seq=0 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff76affd640 1 -- 172.21.15.107:0/573390273 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da2f998180 con 0x55da2fa0fe80 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff76affd640 10 client.16044 handle_command_reply: tid=0 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 client.16044 shutdown 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 msgr2=0x55da2fa06620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009a20 tx=0x7ff77c0b8660 comp rx=0 tx=0).stop 2024-04-24T23:46:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 shutdown_connections 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x55da2fa0fe80 0x55da2fa122a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55da2f993d90 0x55da2fa0dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55da2f98ec20 0x55da2fa06b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff76affd640 0 client.16044 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 --2- 172.21.15.107:0/573390273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55da2f98e230 0x55da2fa06620 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 >> 172.21.15.107:0/573390273 conn(0x55da2f9898a0 msgr2=0x55da2f98d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 shutdown_connections 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 1 -- 172.21.15.107:0/573390273 wait complete. 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff7947e3740 20 client.16044 trim_cache size 0 max 16384 2024-04-24T23:46:58.268 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:58.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 msgr2=0x7ff7780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 0x7ff7780a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff784011cb0 tx=0x7ff78401d000 comp rx=0 tx=0).stop 2024-04-24T23:46:58.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 msgr2=0x7ff78c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009230 tx=0x7ff77c004360 comp rx=0 tx=0).stop 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 shutdown_connections 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ff7780a2b90 0x7ff7780a5050 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ff78c1074c0 0x7ff78c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff78c104b90 0x7ff78c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 --2- 172.21.15.107:0/4013531892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff78c102260 0x7ff78c197b00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 >> 172.21.15.107:0/4013531892 conn(0x7ff78c0fbbf0 msgr2=0x7ff78c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 shutdown_connections 2024-04-24T23:46:58.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.260+0000 7ff791ad4640 1 -- 172.21.15.107:0/4013531892 wait complete. 2024-04-24T23:46:58.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 -- 172.21.15.107:0/2093504009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 msgr2=0x7f0b74106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 --2- 172.21.15.107:0/2093504009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b74106f40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009900 tx=0x7f0b6802a410 comp rx=0 tx=0).stop 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 -- 172.21.15.107:0/2093504009 shutdown_connections 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 --2- 172.21.15.107:0/2093504009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74107480 0x7f0b741098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 --2- 172.21.15.107:0/2093504009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b74106f40 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 --2- 172.21.15.107:0/2093504009 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74102220 0x7f0b74104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 -- 172.21.15.107:0/2093504009 >> 172.21.15.107:0/2093504009 conn(0x7f0b740fbc30 msgr2=0x7f0b740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 -- 172.21.15.107:0/2093504009 shutdown_connections 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 -- 172.21.15.107:0/2093504009 wait complete. 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.371+0000 7f0b79d42640 1 Processor -- start 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 -- start start 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74102220 0x7f0b74101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74107480 0x7f0b74100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7410d430 con 0x7f0b74104b50 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7410d2b0 con 0x7f0b74102220 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b79d42640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7410d5b0 con 0x7f0b74107480 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44570/0 (socket says 172.21.15.107:44570) 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 -- 172.21.15.107:0/1939089500 learned_addr learned my addr 172.21.15.107:0/1939089500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b73fff640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74107480 0x7f0b74100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b737fe640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74102220 0x7f0b74101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 -- 172.21.15.107:0/1939089500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74107480 msgr2=0x7f0b74100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74107480 0x7f0b74100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 -- 172.21.15.107:0/1939089500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74102220 msgr2=0x7f0b74101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74102220 0x7f0b74101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.372+0000 7f0b72ffd640 1 -- 172.21.15.107:0/1939089500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b74100fa0 con 0x7f0b74104b50 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0b68031610 tx=0x7f0b680043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b70ff9640 1 -- 172.21.15.107:0/1939089500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6803f030 con 0x7f0b74104b50 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b70ff9640 1 -- 172.21.15.107:0/1939089500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b68003ce0 con 0x7f0b74104b50 2024-04-24T23:46:58.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b741a1bf0 con 0x7f0b74104b50 2024-04-24T23:46:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b70ff9640 1 -- 172.21.15.107:0/1939089500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68035070 con 0x7f0b74104b50 2024-04-24T23:46:58.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.373+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b741a2140 con 0x7f0b74104b50 2024-04-24T23:46:58.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 Processor -- start 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b70ff9640 1 -- 172.21.15.107:0/1939089500 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0b68042030 con 0x7f0b74104b50 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 -- start start 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b28720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b2b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5610e5a87670 con 0x5610e5b23230 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5610e5a2f970 con 0x5610e5b28d90 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5610e5a83630 con 0x5610e5b23c20 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44578/0 (socket says 172.21.15.107:44578) 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 -- 172.21.15.107:0/4010112267 learned_addr learned my addr 172.21.15.107:0/4010112267 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b2b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b737fe640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b28720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 -- 172.21.15.107:0/4010112267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 msgr2=0x5610e5b28720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b28720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 -- 172.21.15.107:0/4010112267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 msgr2=0x5610e5b2b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b2b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 -- 172.21.15.107:0/4010112267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e5a53400 con 0x5610e5b23230 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b737fe640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b28720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b70ff9640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 0x7f0b5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b73fff640 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0039f0 tx=0x7f0b6c003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27f25d0891e5a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b737fe640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 0x7f0b5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.376+0000 7f0b70ff9640 1 -- 172.21.15.107:0/1939089500 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b680ef9b0 con 0x7f0b74104b50 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b59ffb640 1 -- 172.21.15.107:0/4010112267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c01a470 con 0x5610e5b23230 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b59ffb640 1 -- 172.21.15.107:0/4010112267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b6c015070 con 0x5610e5b23230 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b59ffb640 1 -- 172.21.15.107:0/4010112267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c010900 con 0x5610e5b23230 2024-04-24T23:46:58.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/4010112267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 msgr2=0x5610e5b23650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0039f0 tx=0x7f0b6c003eb0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b737fe640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 0x7f0b5c0a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24020 tx=0x7f0b6400c700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/4010112267 shutdown_connections 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b2b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b28720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/4010112267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b23650 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/4010112267 >> 172.21.15.107:0/4010112267 conn(0x5610e5b1e8a0 msgr2=0x5610e5b1eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/4010112267 shutdown_connections 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.377+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/4010112267 wait complete. 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 -- start start 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 Processor -- start 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23230 0x5610e5b2d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b34910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5610e5b2bd40 con 0x5610e5b23c20 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5610e5a43050 con 0x5610e5b28d90 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5610e5b2bec0 con 0x5610e5b23230 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b7ca4d740 10 client.0 _ll_register_callbacks cb 0x5610e579abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44584/0 (socket says 172.21.15.107:44584) 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 -- 172.21.15.107:0/596944939 learned_addr learned my addr 172.21.15.107:0/596944939 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b73fff640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b34910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 -- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23230 msgr2=0x5610e5b2d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b737fe640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23230 0x5610e5b2d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23230 0x5610e5b2d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 -- 172.21.15.107:0/596944939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 msgr2=0x5610e5b34910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b34910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.378+0000 7f0b72ffd640 1 -- 172.21.15.107:0/596944939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e5a14dc0 con 0x5610e5b23c20 2024-04-24T23:46:58.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b8810 tx=0x7f0b6803b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1914c980e3cd3a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b680efe60 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b6803bd70 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b6c015070 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b68049050 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5610e579b3f0 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b6c015070 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0b68003800 con 0x5610e5b23c20 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 10 client.16056 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 20 client.16056 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5b390f0 0x5610e5b3b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 4 client.16056 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.379+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610e5a94bb0 con 0x5610e5b390f0 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.380+0000 7f0b737fe640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5b390f0 0x5610e5b3b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.380+0000 7f0b52ffd640 10 client.16056 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.380+0000 7f0b737fe640 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5b390f0 0x5610e5b3b590 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dc83a3129a834f2d server_cookie=b8881264f8c1caab in_seq=0 out_seq=0 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b52ffd640 1 -- 172.21.15.107:0/596944939 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610e5a94bb0 con 0x5610e5b390f0 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b52ffd640 10 client.16056 handle_command_reply: tid=0 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 client.16056 shutdown 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 msgr2=0x5610e5b2d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b8810 tx=0x7f0b6803b870 comp rx=0 tx=0).stop 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 shutdown_connections 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5b390f0 0x5610e5b3b590 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b28d90 0x5610e5b34910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23c20 0x5610e5b2d8d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/596944939 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23230 0x5610e5b2d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b52ffd640 0 client.16056 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 >> 172.21.15.107:0/596944939 conn(0x5610e5b1e8a0 msgr2=0x5610e5b1db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 shutdown_connections 2024-04-24T23:46:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/596944939 wait complete. 2024-04-24T23:46:58.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.381+0000 7f0b7ca4d740 20 client.16056 trim_cache size 0 max 16384 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 Processor -- start 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 -- start start 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 0x5610e5b348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b28d90 0x5610e5b3fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741ac690 con 0x5610e5b23230 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.394+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741ae1d0 con 0x5610e5b23c20 2024-04-24T23:46:58.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741acd60 con 0x5610e5b28d90 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44586/0 (socket says 172.21.15.107:44586) 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 -- 172.21.15.107:0/879901867 learned_addr learned my addr 172.21.15.107:0/879901867 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b73fff640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b28d90 0x5610e5b3fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 0x5610e5b348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 -- 172.21.15.107:0/879901867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b28d90 msgr2=0x5610e5b3fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b28d90 0x5610e5b3fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 -- 172.21.15.107:0/879901867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 msgr2=0x5610e5b348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 0x5610e5b348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 -- 172.21.15.107:0/879901867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e58657d0 con 0x5610e5b23230 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 0x5610e5b348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b737fe640 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24690 tx=0x7f0b64013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8292757db0778041 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b597fa640 1 -- 172.21.15.107:0/879901867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64027330 con 0x5610e5b23230 2024-04-24T23:46:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b597fa640 1 -- 172.21.15.107:0/879901867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b6401e420 con 0x5610e5b23230 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.395+0000 7f0b597fa640 1 -- 172.21.15.107:0/879901867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6401d670 con 0x5610e5b23230 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/879901867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 msgr2=0x5610e5b1dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24690 tx=0x7f0b64013000 comp rx=0 tx=0).stop 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/879901867 shutdown_connections 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b28d90 0x5610e5b3fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23c20 0x5610e5b348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/879901867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b23230 0x5610e5b1dd80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/879901867 >> 172.21.15.107:0/879901867 conn(0x5610e5b1e8a0 msgr2=0x5610e5b24f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/879901867 shutdown_connections 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/879901867 wait complete. 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- start start 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 Processor -- start 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 0x5610e5b32670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b32bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741adb00 con 0x5610e5b28d90 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741ad430 con 0x5610e5b23230 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.396+0000 7f0b7ca4d740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b741aa670 con 0x5610e5b23c20 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b7ca4d740 10 client.0 _ll_register_callbacks cb 0x7f0b68005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44592/0 (socket says 172.21.15.107:44592) 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 -- 172.21.15.107:0/2011160674 learned_addr learned my addr 172.21.15.107:0/2011160674 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b737fe640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 0x5610e5b32670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b72ffd640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b32bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 -- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 msgr2=0x5610e5b32bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b32bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 -- 172.21.15.107:0/2011160674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 msgr2=0x5610e5b32670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 0x5610e5b32670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 -- 172.21.15.107:0/2011160674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e5b341b0 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b737fe640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 0x5610e5b32670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b73fff640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24480 tx=0x7f0b6c005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7362f64cfed7d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c02d060 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b6401e420 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b6c011850 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b6c034050 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b7ca4d740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.397+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5610e5b9fd00 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b6401e420 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0b6c01abe0 con 0x5610e5b28d90 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 20 client.16062 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5ba04f0 0x5610e5ba2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 4 client.16062 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610e5a8ef00 con 0x5610e5ba04f0 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b737fe640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5ba04f0 0x5610e5ba2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b52ffd640 10 client.16062 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.398+0000 7f0b737fe640 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5ba04f0 0x5610e5ba2910 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=47a0682fc9afbc4c server_cookie=6c340087634f34e7 in_seq=0 out_seq=0 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b52ffd640 1 -- 172.21.15.107:0/2011160674 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610e5a8ef00 con 0x5610e5ba04f0 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b52ffd640 10 client.16062 handle_command_reply: tid=0 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 client.16062 shutdown 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 msgr2=0x5610e5b33bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24480 tx=0x7f0b6c005980 comp rx=0 tx=0).stop 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 shutdown_connections 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5610e5ba04f0 0x5610e5ba2910 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5610e5b28d90 0x5610e5b33bc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b52ffd640 0 client.16062 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5610e5b23c20 0x5610e5b32bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 --2- 172.21.15.107:0/2011160674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5610e5b23230 0x5610e5b32670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 >> 172.21.15.107:0/2011160674 conn(0x5610e5b1e8a0 msgr2=0x5610e5b246c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 shutdown_connections 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 1 -- 172.21.15.107:0/2011160674 wait complete. 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b7ca4d740 20 client.16062 trim_cache size 0 max 16384 2024-04-24T23:46:58.408 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 msgr2=0x7f0b5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 0x7f0b5c0a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5610e5b24020 tx=0x7f0b6400c700 comp rx=0 tx=0).stop 2024-04-24T23:46:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 msgr2=0x7f0b741001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.399+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0b68031610 tx=0x7f0b680043b0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 shutdown_connections 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b5c0a2b90 0x7f0b5c0a5050 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b74107480 0x7f0b74100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b74104b50 0x7f0b741001d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 --2- 172.21.15.107:0/1939089500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b74102220 0x7f0b74101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 >> 172.21.15.107:0/1939089500 conn(0x7f0b740fbc30 msgr2=0x7f0b74102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 shutdown_connections 2024-04-24T23:46:58.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.400+0000 7f0b79d42640 1 -- 172.21.15.107:0/1939089500 wait complete. 2024-04-24T23:46:58.410 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:58 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1905034720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:58.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T23:46:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 -- 172.21.15.107:0/3910338389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c101890 msgr2=0x7f938c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 --2- 172.21.15.107:0/3910338389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c101890 0x7f938c101d10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f937c009900 tx=0x7f937c02a410 comp rx=0 tx=0).stop 2024-04-24T23:46:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 -- 172.21.15.107:0/3910338389 shutdown_connections 2024-04-24T23:46:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 --2- 172.21.15.107:0/3910338389 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c102250 0x7f938c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 --2- 172.21.15.107:0/3910338389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c101890 0x7f938c101d10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 --2- 172.21.15.107:0/3910338389 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938c1006a0 0x7f938c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 -- 172.21.15.107:0/3910338389 >> 172.21.15.107:0/3910338389 conn(0x7f938c0fbc30 msgr2=0x7f938c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 -- 172.21.15.107:0/3910338389 shutdown_connections 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.512+0000 7f9393d14640 1 -- 172.21.15.107:0/3910338389 wait complete. 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 Processor -- start 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 -- start start 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c101890 0x7f938c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938c102250 0x7f938c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f938c10c320 con 0x7f938c1006a0 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f938c10c1a0 con 0x7f938c102250 2024-04-24T23:46:58.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9393d14640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f938c10c4a0 con 0x7f938c101890 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44618/0 (socket says 172.21.15.107:44618) 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 -- 172.21.15.107:0/2196263568 learned_addr learned my addr 172.21.15.107:0/2196263568 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 -- 172.21.15.107:0/2196263568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c101890 msgr2=0x7f938c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c101890 0x7f938c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 -- 172.21.15.107:0/2196263568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938c102250 msgr2=0x7f938c19df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938c102250 0x7f938c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.513+0000 7f9391ab0640 1 -- 172.21.15.107:0/2196263568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c19e5e0 con 0x7f938c1006a0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f938800e9c0 tx=0x7f938800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9382ffd640 1 -- 172.21.15.107:0/2196263568 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938800cd80 con 0x7f938c1006a0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9382ffd640 1 -- 172.21.15.107:0/2196263568 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93880035f0 con 0x7f938c1006a0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938c19e8d0 con 0x7f938c1006a0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9382ffd640 1 -- 172.21.15.107:0/2196263568 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388010690 con 0x7f938c1006a0 2024-04-24T23:46:58.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.514+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938c19ee70 con 0x7f938c1006a0 2024-04-24T23:46:58.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.516+0000 7f9382ffd640 1 -- 172.21.15.107:0/2196263568 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f93880037a0 con 0x7f938c1006a0 2024-04-24T23:46:58.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.516+0000 7f9396a27740 1 Processor -- start 2024-04-24T23:46:58.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 -- start start 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4537860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 0x556cf453c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 0x556cf453f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cf4499670 con 0x556cf4537e30 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556cf4441970 con 0x556cf453cfa0 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9396a27740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x556cf4495630 con 0x556cf4537440 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9391ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 0x556cf453c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9382ffd640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 0x7f93780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9391ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 0x556cf453c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44630/0 (socket says 172.21.15.107:44630) 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9391ab0640 1 -- 172.21.15.107:0/1114585661 learned_addr learned my addr 172.21.15.107:0/1114585661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 0x556cf453f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93922b1640 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4537860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93922b1640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 0x7f93780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 -- 172.21.15.107:0/1114585661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 msgr2=0x556cf4537860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4537860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 -- 172.21.15.107:0/1114585661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 msgr2=0x556cf453c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f9382ffd640 1 -- 172.21.15.107:0/2196263568 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f93880cf930 con 0x7f938c1006a0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 0x556cf453c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.517+0000 7f93912af640 1 -- 172.21.15.107:0/1114585661 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf4465400 con 0x556cf453cfa0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.518+0000 7f93922b1640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 0x7f93780a50d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9384009270 tx=0x7f9384005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.518+0000 7f93912af640 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 0x556cf453f430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f937c02a930 tx=0x7f937c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2dc28c51c8400b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.518+0000 7f9377fff640 1 -- 172.21.15.107:0/1114585661 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c03e030 con 0x556cf453cfa0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.518+0000 7f9377fff640 1 -- 172.21.15.107:0/1114585661 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c004040 con 0x556cf453cfa0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.518+0000 7f9377fff640 1 -- 172.21.15.107:0/1114585661 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c035040 con 0x556cf453cfa0 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- 172.21.15.107:0/1114585661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 msgr2=0x556cf453f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 0x556cf453f430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f937c02a930 tx=0x7f937c000bc0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- 172.21.15.107:0/1114585661 shutdown_connections 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf453cfa0 0x556cf453f430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf4537e30 0x556cf453c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- 172.21.15.107:0/1114585661 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4537860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- 172.21.15.107:0/1114585661 >> 172.21.15.107:0/1114585661 conn(0x556cf4532ab0 msgr2=0x556cf4532eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- 172.21.15.107:0/1114585661 shutdown_connections 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- 172.21.15.107:0/1114585661 wait complete. 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- start start 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 Processor -- start 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf4541540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4541a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cf453fef0 con 0x556cf453cfa0 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556cf4455050 con 0x556cf4537440 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x556cf4540070 con 0x556cf4537e30 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.519+0000 7f9396a27740 10 client.0 _ll_register_callbacks cb 0x556cf41acbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44640/0 (socket says 172.21.15.107:44640) 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 -- 172.21.15.107:0/2110242929 learned_addr learned my addr 172.21.15.107:0/2110242929 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf4541540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93912af640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4541a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 -- 172.21.15.107:0/2110242929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 msgr2=0x556cf4541a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4541a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 -- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 msgr2=0x556cf4541540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf4541540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 -- 172.21.15.107:0/2110242929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf4426dc0 con 0x556cf453cfa0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93922b1640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f938400fb10 tx=0x7f9384004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da2f426740bad46 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93757fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384025070 con 0x556cf453cfa0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9384011dd0 con 0x556cf453cfa0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937c004040 con 0x556cf453cfa0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9384030050 con 0x556cf453cfa0 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f9396a27740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cf41ad3f0 con 0x556cf453cfa0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.520+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c004040 con 0x556cf453cfa0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f938401f420 con 0x556cf453cfa0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 10 client.16074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 20 client.16074 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf454d420 0x556cf454f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 4 client.16074 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cf44a6bb0 con 0x556cf454d420 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf454d420 0x556cf454f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f93757fa640 10 client.16074 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.521+0000 7f9391ab0640 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf454d420 0x556cf454f8c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=161a60bb63204106 server_cookie=e9ec57588f074b2a in_seq=0 out_seq=0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f93757fa640 1 -- 172.21.15.107:0/2110242929 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cf44a6bb0 con 0x556cf454d420 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f93757fa640 10 client.16074 handle_command_reply: tid=0 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 client.16074 shutdown 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 msgr2=0x556cf4548ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f938400fb10 tx=0x7f9384004a40 comp rx=0 tx=0).stop 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 shutdown_connections 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf454d420 0x556cf454f8c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548ac0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f93757fa640 0 client.16074 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4541a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 --2- 172.21.15.107:0/2110242929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf4541540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 >> 172.21.15.107:0/2110242929 conn(0x556cf4532ab0 msgr2=0x556cf452fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 shutdown_connections 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.522+0000 7f9396a27740 1 -- 172.21.15.107:0/2110242929 wait complete. 2024-04-24T23:46:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.523+0000 7f9396a27740 20 client.16074 trim_cache size 0 max 16384 2024-04-24T23:46:58.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.535+0000 7f9396a27740 1 Processor -- start 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.535+0000 7f9396a27740 1 -- start start 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4538fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537e30 0x556cf4539520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a82a0 con 0x556cf453cfa0 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a9de0 con 0x556cf4537e30 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f9396a27740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a8970 con 0x556cf4537440 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44650/0 (socket says 172.21.15.107:44650) 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 -- 172.21.15.107:0/4268335010 learned_addr learned my addr 172.21.15.107:0/4268335010 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 -- 172.21.15.107:0/4268335010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 msgr2=0x556cf4538fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4538fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 -- 172.21.15.107:0/4268335010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537e30 msgr2=0x556cf4539520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537e30 0x556cf4539520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 -- 172.21.15.107:0/4268335010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf42777d0 con 0x556cf453cfa0 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93922b1640 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9384010010 tx=0x7f93840148f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb8d1ac80ba5e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93777fe640 1 -- 172.21.15.107:0/4268335010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384039070 con 0x556cf453cfa0 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93777fe640 1 -- 172.21.15.107:0/4268335010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9384009c10 con 0x556cf453cfa0 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.536+0000 7f93777fe640 1 -- 172.21.15.107:0/4268335010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384025040 con 0x556cf453cfa0 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- 172.21.15.107:0/4268335010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 msgr2=0x556cf4548a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9384010010 tx=0x7f93840148f0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- 172.21.15.107:0/4268335010 shutdown_connections 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4548a80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537e30 0x556cf4539520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- 172.21.15.107:0/4268335010 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537440 0x556cf4538fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- 172.21.15.107:0/4268335010 >> 172.21.15.107:0/4268335010 conn(0x556cf4532ab0 msgr2=0x556cf45388d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- 172.21.15.107:0/4268335010 shutdown_connections 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- 172.21.15.107:0/4268335010 wait complete. 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- start start 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 Processor -- start 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf45488d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4546ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a9710 con 0x556cf453cfa0 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a9040 con 0x556cf4537440 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a6280 con 0x556cf4537e30 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.537+0000 7f9396a27740 10 client.0 _ll_register_callbacks cb 0x7f9384014f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44660/0 (socket says 172.21.15.107:44660) 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 -- 172.21.15.107:0/334431566 learned_addr learned my addr 172.21.15.107:0/334431566 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93912af640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4546ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f9391ab0640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf45488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 -- 172.21.15.107:0/334431566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 msgr2=0x556cf4546ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4546ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 -- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 msgr2=0x556cf45488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf45488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 -- 172.21.15.107:0/334431566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf4547b10 con 0x556cf453cfa0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f9391ab0640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf45488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93912af640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4546ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93922b1640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9384037040 tx=0x7f938401f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2e55eda760bf07 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93757fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384039070 con 0x556cf453cfa0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9384009c10 con 0x556cf453cfa0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9384014040 con 0x556cf453cfa0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9384025040 con 0x556cf453cfa0 2024-04-24T23:46:58.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f9396a27740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cf45b4940 con 0x556cf453cfa0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.538+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9384009c10 con 0x556cf453cfa0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9384034430 con 0x556cf453cfa0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 10 client.16080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 20 client.16080 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf45b51e0 0x556cf45b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 4 client.16080 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cf44a0f00 con 0x556cf45b51e0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9391ab0640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf45b51e0 0x556cf45b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f93757fa640 10 client.16080 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9391ab0640 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf45b51e0 0x556cf45b75d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec71cf7116108e55 server_cookie=38fc23fb7f4cd6f3 in_seq=0 out_seq=0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f93757fa640 1 -- 172.21.15.107:0/334431566 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cf44a0f00 con 0x556cf45b51e0 2024-04-24T23:46:58.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f93757fa640 10 client.16080 handle_command_reply: tid=0 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 1 client.16080 shutdown 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 msgr2=0x556cf4547420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.539+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9384037040 tx=0x7f938401f6d0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 shutdown_connections 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x556cf45b51e0 0x556cf45b75d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf453cfa0 0x556cf4547420 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x556cf4537e30 0x556cf4546ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 --2- 172.21.15.107:0/334431566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556cf4537440 0x556cf45488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f93757fa640 0 client.16080 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 >> 172.21.15.107:0/334431566 conn(0x556cf4532ab0 msgr2=0x556cf4536250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 shutdown_connections 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 1 -- 172.21.15.107:0/334431566 wait complete. 2024-04-24T23:46:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9396a27740 20 client.16080 trim_cache size 0 max 16384 2024-04-24T23:46:58.547 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 msgr2=0x7f93780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 0x7f93780a50d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9384009270 tx=0x7f9384005bd0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 msgr2=0x7f938c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f938800e9c0 tx=0x7f938800ee80 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 shutdown_connections 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f93780a2c10 0x7f93780a50d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938c102250 0x7f938c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f938c101890 0x7f938c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 --2- 172.21.15.107:0/2196263568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c1006a0 0x7f938c1969c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 >> 172.21.15.107:0/2196263568 conn(0x7f938c0fbc30 msgr2=0x7f938c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 shutdown_connections 2024-04-24T23:46:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.540+0000 7f9393d14640 1 -- 172.21.15.107:0/2196263568 wait complete. 2024-04-24T23:46:58.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T23:46:58.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:58 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1905034720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:58.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:58 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1905034720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 -- 172.21.15.107:0/2009872858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 msgr2=0x7ffbc4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/2009872858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4101bb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009900 tx=0x7ffbc002a410 comp rx=0 tx=0).stop 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 -- 172.21.15.107:0/2009872858 shutdown_connections 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/2009872858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc4103340 0x7ffbc4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/2009872858 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbc4102980 0x7ffbc4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/2009872858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4101bb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 -- 172.21.15.107:0/2009872858 >> 172.21.15.107:0/2009872858 conn(0x7ffbc40fcd20 msgr2=0x7ffbc40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 -- 172.21.15.107:0/2009872858 shutdown_connections 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.650+0000 7ffbcb45d640 1 -- 172.21.15.107:0/2009872858 wait complete. 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 Processor -- start 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 -- start start 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbc4102980 0x7ffbc4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc4103340 0x7ffbc419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc410d400 con 0x7ffbc4101790 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc410d280 con 0x7ffbc4103340 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc410d580 con 0x7ffbc4102980 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44682/0 (socket says 172.21.15.107:44682) 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4142349125 learned_addr learned my addr 172.21.15.107:0/4142349125 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4142349125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbc4102980 msgr2=0x7ffbc4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbc4102980 0x7ffbc4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4142349125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc4103340 msgr2=0x7ffbc419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc4103340 0x7ffbc419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4142349125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc419f720 con 0x7ffbc4101790 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0031610 tx=0x7ffbc0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbba7fc640 1 -- 172.21.15.107:0/4142349125 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc003f030 con 0x7ffbc4101790 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbba7fc640 1 -- 172.21.15.107:0/4142349125 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc0003ad0 con 0x7ffbc4101790 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc419f9b0 con 0x7ffbc4101790 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.651+0000 7ffbba7fc640 1 -- 172.21.15.107:0/4142349125 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0035070 con 0x7ffbc4101790 2024-04-24T23:46:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.652+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc419fea0 con 0x7ffbc4101790 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbba7fc640 1 -- 172.21.15.107:0/4142349125 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ffbc0042030 con 0x7ffbc4101790 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 Processor -- start 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 -- start start 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e7e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e86220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562f76de2670 con 0x562f76e7ec20 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562f76d8a970 con 0x562f76e83d90 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbce169740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562f76dde630 con 0x562f76e7e230 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbba7fc640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 0x7ffbb00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44692/0 (socket says 172.21.15.107:44692) 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3246478725 learned_addr learned my addr 172.21.15.107:0/3246478725 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e86220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.654+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 0x7ffbb00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e7e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbba7fc640 1 -- 172.21.15.107:0/4142349125 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffbc00efc40 con 0x7ffbc4101790 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3246478725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 msgr2=0x562f76e7e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e7e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3246478725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 msgr2=0x562f76e86220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e86220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3246478725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f76dae400 con 0x562f76e7ec20 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b8530 tx=0x7ffbc00bbfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72371f3238e79980 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 0x7ffbb00a5050 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x562f76e7f480 tx=0x7ffbbc0094b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbaf7fe640 1 -- 172.21.15.107:0/3246478725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0049030 con 0x562f76e7ec20 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbaf7fe640 1 -- 172.21.15.107:0/3246478725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00bc7d0 con 0x562f76e7ec20 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.655+0000 7ffbaf7fe640 1 -- 172.21.15.107:0/3246478725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0003c80 con 0x562f76e7ec20 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- 172.21.15.107:0/3246478725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 msgr2=0x562f76e83720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b8530 tx=0x7ffbc00bbfd0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- 172.21.15.107:0/3246478725 shutdown_connections 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e86220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e83720 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- 172.21.15.107:0/3246478725 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e7e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- 172.21.15.107:0/3246478725 >> 172.21.15.107:0/3246478725 conn(0x562f76e798a0 msgr2=0x562f76e79ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- 172.21.15.107:0/3246478725 shutdown_connections 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- 172.21.15.107:0/3246478725 wait complete. 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- start start 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 Processor -- start 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e8f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562f76de2670 con 0x562f76e7e230 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.656+0000 7ffbce169740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562f76d9e050 con 0x562f76e83d90 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbce169740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x562f76e86b00 con 0x562f76e7ec20 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbce169740 10 client.0 _ll_register_callbacks cb 0x562f76af5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44698/0 (socket says 172.21.15.107:44698) 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3812195410 learned_addr learned my addr 172.21.15.107:0/3812195410 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e8f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 msgr2=0x562f76e884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e884d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3812195410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 msgr2=0x562f76e8f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e8f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 -- 172.21.15.107:0/3812195410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f76d6fdc0 con 0x562f76e7e230 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00ed3c0 tx=0x7ffbc00bc0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bc3c145928eafc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbacff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0049070 con 0x562f76e7e230 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffbc00b8740 con 0x562f76e7e230 2024-04-24T23:46:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc00bc7d0 con 0x562f76e7e230 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc00d50c0 con 0x562f76e7e230 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbce169740 10 client.16092 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f76af63f0 con 0x562f76e7e230 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.657+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00bc7d0 con 0x562f76e7e230 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffbc00bc330 con 0x562f76e7e230 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 10 client.16092 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 20 client.16092 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76e93ac0 0x562f76e95eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 4 client.16092 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f76defbb0 con 0x562f76e93ac0 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76e93ac0 0x562f76e95eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbacff9640 10 client.16092 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.658+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76e93ac0 0x562f76e95eb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=311201ea663042eb server_cookie=bc9565f5f65f29bd in_seq=0 out_seq=0 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbacff9640 1 -- 172.21.15.107:0/3812195410 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f76defbb0 con 0x562f76e93ac0 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbacff9640 10 client.16092 handle_command_reply: tid=0 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 client.16092 shutdown 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 msgr2=0x562f76e87f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00ed3c0 tx=0x7ffbc00bc0d0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 shutdown_connections 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76e93ac0 0x562f76e95eb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e8f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbacff9640 0 client.16092 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 --2- 172.21.15.107:0/3812195410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e87f90 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 >> 172.21.15.107:0/3812195410 conn(0x562f76e798a0 msgr2=0x562f76e78b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 shutdown_connections 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.659+0000 7ffbce169740 1 -- 172.21.15.107:0/3812195410 wait complete. 2024-04-24T23:46:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.660+0000 7ffbce169740 20 client.16092 trim_cache size 0 max 16384 2024-04-24T23:46:58.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 Processor -- start 2024-04-24T23:46:58.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 -- start start 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e8f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e789c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41aa4f0 con 0x562f76e7e230 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41ac030 con 0x562f76e83d90 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbce169740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41aabc0 con 0x562f76e7ec20 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44712/0 (socket says 172.21.15.107:44712) 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4292034298 learned_addr learned my addr 172.21.15.107:0/4292034298 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e8f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4292034298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 msgr2=0x562f76e8f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e8f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4292034298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 msgr2=0x562f76e789c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e789c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 -- 172.21.15.107:0/4292034298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f76e96220 con 0x562f76e7e230 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e8f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b8530 tx=0x7ffbc00bbfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f104c2e51b1f4e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbaeffd640 1 -- 172.21.15.107:0/4292034298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00bbc10 con 0x562f76e7e230 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbaeffd640 1 -- 172.21.15.107:0/4292034298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00d50c0 con 0x562f76e7e230 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.673+0000 7ffbaeffd640 1 -- 172.21.15.107:0/4292034298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0003c80 con 0x562f76e7e230 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- 172.21.15.107:0/4292034298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 msgr2=0x562f76e88430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00b8530 tx=0x7ffbc00bbfd0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- 172.21.15.107:0/4292034298 shutdown_connections 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76e789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7ec20 0x562f76e8f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 --2- 172.21.15.107:0/4292034298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7e230 0x562f76e88430 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- 172.21.15.107:0/4292034298 >> 172.21.15.107:0/4292034298 conn(0x562f76e798a0 msgr2=0x562f76e782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- 172.21.15.107:0/4292034298 shutdown_connections 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- 172.21.15.107:0/4292034298 wait complete. 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 -- start start 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.674+0000 7ffbce169740 1 Processor -- start 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e8c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76ef7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41ab960 con 0x562f76e7ec20 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41ab290 con 0x562f76e83d90 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc41a84d0 con 0x562f76e7e230 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 10 client.0 _ll_register_callbacks cb 0x7ffbb4009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44716/0 (socket says 172.21.15.107:44716) 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 -- 172.21.15.107:0/2889669664 learned_addr learned my addr 172.21.15.107:0/2889669664 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e8c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc99fa640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76ef7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 -- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 msgr2=0x562f76e8c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e8c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 -- 172.21.15.107:0/2889669664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 msgr2=0x562f76ef7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76ef7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 -- 172.21.15.107:0/2889669664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f76bc07d0 con 0x562f76e7ec20 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbc89f8640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x562f76e7a770 tx=0x7ffbb40044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f79c1cda77020d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbacff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4020070 con 0x562f76e7ec20 2024-04-24T23:46:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffbb401c310 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.675+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc00d50c0 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbb402c050 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 10 client.16098 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f76ef97a0 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00d50c0 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffbb4010c50 con 0x562f76e7ec20 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 20 client.16098 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76efa150 0x562f76efc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 4 client.16098 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f76de9f00 con 0x562f76efa150 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.676+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76efa150 0x562f76efc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbc91f9640 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76efa150 0x562f76efc570 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85f8eb011a37c641 server_cookie=6f5c11a34da57e3c in_seq=0 out_seq=0 2024-04-24T23:46:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbacff9640 10 client.16098 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbacff9640 1 -- 172.21.15.107:0/2889669664 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f76de9f00 con 0x562f76efa150 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbacff9640 10 client.16098 handle_command_reply: tid=0 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 client.16098 shutdown 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 msgr2=0x562f76e8ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x562f76e7a770 tx=0x7ffbb40044d0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 shutdown_connections 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x562f76efa150 0x562f76efc570 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562f76e83d90 0x562f76ef7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbacff9640 0 client.16098 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f76e7ec20 0x562f76e8ca70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 --2- 172.21.15.107:0/2889669664 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x562f76e7e230 0x562f76e8c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 >> 172.21.15.107:0/2889669664 conn(0x562f76e798a0 msgr2=0x562f76e88290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 shutdown_connections 2024-04-24T23:46:58.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 1 -- 172.21.15.107:0/2889669664 wait complete. 2024-04-24T23:46:58.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.677+0000 7ffbce169740 20 client.16098 trim_cache size 0 max 16384 2024-04-24T23:46:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 msgr2=0x7ffbb00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 0x7ffbb00a5050 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x562f76e7f480 tx=0x7ffbbc0094b0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 msgr2=0x7ffbc4197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0031610 tx=0x7ffbc0004270 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 shutdown_connections 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffbb00a2b90 0x7ffbb00a5050 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc4103340 0x7ffbc419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffbc4102980 0x7ffbc4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 --2- 172.21.15.107:0/4142349125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbc4101790 0x7ffbc4197b00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 >> 172.21.15.107:0/4142349125 conn(0x7ffbc40fcd20 msgr2=0x7ffbc4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 shutdown_connections 2024-04-24T23:46:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.678+0000 7ffbcb45d640 1 -- 172.21.15.107:0/4142349125 wait complete. 2024-04-24T23:46:58.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 -- 172.21.15.107:0/308130814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50107480 msgr2=0x7f0b501098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/308130814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50107480 0x7f0b501098d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009ca0 tx=0x7f0b4802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 -- 172.21.15.107:0/308130814 shutdown_connections 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/308130814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50107480 0x7f0b501098d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/308130814 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50104b50 0x7f0b50106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/308130814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50102220 0x7f0b50104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 -- 172.21.15.107:0/308130814 >> 172.21.15.107:0/308130814 conn(0x7f0b500fbc30 msgr2=0x7f0b500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 -- 172.21.15.107:0/308130814 shutdown_connections 2024-04-24T23:46:58.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.791+0000 7f0b54ee1640 1 -- 172.21.15.107:0/308130814 wait complete. 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 Processor -- start 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 -- start start 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50104b50 0x7f0b50198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50107480 0x7f0b5019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d440 con 0x7f0b50102220 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d2c0 con 0x7f0b50104b50 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b54ee1640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d5c0 con 0x7f0b50107480 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44742/0 (socket says 172.21.15.107:44742) 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3177153132 learned_addr learned my addr 172.21.15.107:0/3177153132 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50107480 0x7f0b5019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.792+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50104b50 0x7f0b50198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3177153132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50107480 msgr2=0x7f0b5019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50107480 0x7f0b5019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3177153132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50104b50 msgr2=0x7f0b50198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50104b50 0x7f0b50198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3177153132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5019f6e0 con 0x7f0b50102220 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000d950 tx=0x7f0b4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b47fff640 1 -- 172.21.15.107:0/3177153132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40017030 con 0x7f0b50102220 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b47fff640 1 -- 172.21.15.107:0/3177153132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b40004020 con 0x7f0b50102220 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b5019f9d0 con 0x7f0b50102220 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b47fff640 1 -- 172.21.15.107:0/3177153132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b400047a0 con 0x7f0b50102220 2024-04-24T23:46:58.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.793+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b5019ff20 con 0x7f0b50102220 2024-04-24T23:46:58.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.796+0000 7f0b47fff640 1 -- 172.21.15.107:0/3177153132 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0b40004940 con 0x7f0b50102220 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.796+0000 7f0b57bf4740 1 Processor -- start 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.796+0000 7f0b57bf4740 1 -- start start 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c326f030 0x5593c3273b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c32741a0 0x5593c3276630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5593c31ce670 con 0x5593c326e640 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5593c3176970 con 0x5593c326f030 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5593c31ca630 con 0x5593c32741a0 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44746/0 (socket says 172.21.15.107:44746) 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 -- 172.21.15.107:0/4159714107 learned_addr learned my addr 172.21.15.107:0/4159714107 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c326f030 0x5593c3273b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b47fff640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 0x7f0b3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c32741a0 0x5593c3276630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 -- 172.21.15.107:0/4159714107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c32741a0 msgr2=0x5593c3276630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c32741a0 0x5593c3276630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 -- 172.21.15.107:0/4159714107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c326f030 msgr2=0x5593c3273b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c326f030 0x5593c3273b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 -- 172.21.15.107:0/4159714107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c319a400 con 0x5593c326e640 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b47fff640 1 -- 172.21.15.107:0/3177153132 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b400cee10 con 0x7f0b50102220 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 0x7f0b3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.797+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0b48035000 tx=0x7f0b48005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb80f5bfb1138d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b44d99640 1 -- 172.21.15.107:0/4159714107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48042040 con 0x5593c326e640 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b44d99640 1 -- 172.21.15.107:0/4159714107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b48044070 con 0x5593c326e640 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b44d99640 1 -- 172.21.15.107:0/4159714107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48052440 con 0x5593c326e640 2024-04-24T23:46:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 0x7f0b3c0a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0b4804d000 tx=0x7f0b4804e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 -- 172.21.15.107:0/4159714107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 msgr2=0x5593c326ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0b48035000 tx=0x7f0b48005f40 comp rx=0 tx=0).stop 2024-04-24T23:46:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 -- 172.21.15.107:0/4159714107 shutdown_connections 2024-04-24T23:46:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c32741a0 0x5593c3276630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c326f030 0x5593c3273b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/4159714107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c326ea60 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.798+0000 7f0b57bf4740 1 -- 172.21.15.107:0/4159714107 >> 172.21.15.107:0/4159714107 conn(0x5593c32658a0 msgr2=0x5593c3265ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- 172.21.15.107:0/4159714107 shutdown_connections 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- 172.21.15.107:0/4159714107 wait complete. 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- start start 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 Processor -- start 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326e640 0x5593c32787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5593c3277150 con 0x5593c326f030 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5593c318a050 con 0x5593c32741a0 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5593c32772d0 con 0x5593c326e640 2024-04-24T23:46:58.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b57bf4740 10 client.0 _ll_register_callbacks cb 0x5593c2ee1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b45d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b45d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44750/0 (socket says 172.21.15.107:44750) 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.799+0000 7f0b45d9b640 1 -- 172.21.15.107:0/3903291263 learned_addr learned my addr 172.21.15.107:0/3903291263 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326e640 0x5593c32787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 -- 172.21.15.107:0/3903291263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326e640 msgr2=0x5593c32787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326e640 0x5593c32787e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 -- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 msgr2=0x5593c327fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 -- 172.21.15.107:0/3903291263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c315bdc0 con 0x5593c326f030 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0b38012700 tx=0x7f0b38012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105571bcf1e6ea0a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38004050 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b380041f0 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b48044070 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b380043f0 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b57bf4740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5593c2ee23f0 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.800+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b48044070 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0b38003af0 con 0x5593c326f030 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 20 client.16110 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c3284640 0x5593c3286ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 4 client.16110 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593c31dbbb0 con 0x5593c3284640 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c3284640 0x5593c3286ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b35ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.801+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c3284640 0x5593c3286ac0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c7246e4b37d6fd server_cookie=632fcd93a205951f in_seq=0 out_seq=0 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3903291263 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593c31dbbb0 con 0x5593c3284640 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b35ffb640 10 client.16110 handle_command_reply: tid=0 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 client.16110 shutdown 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 msgr2=0x5593c3278d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0b38012700 tx=0x7f0b38012bc0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 shutdown_connections 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c3284640 0x5593c3286ac0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326f030 0x5593c3278d20 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3903291263 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326e640 0x5593c32787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b35ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.802+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 >> 172.21.15.107:0/3903291263 conn(0x5593c32658a0 msgr2=0x5593c3264b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.803+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 shutdown_connections 2024-04-24T23:46:58.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.803+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3903291263 wait complete. 2024-04-24T23:46:58.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.803+0000 7f0b57bf4740 20 client.16110 trim_cache size 0 max 16384 2024-04-24T23:46:58.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 Processor -- start 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 -- start start 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c326f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a8350 con 0x5593c326e640 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a9e90 con 0x5593c32741a0 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a8a20 con 0x5593c326f030 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.816+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44760/0 (socket says 172.21.15.107:44760) 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 -- 172.21.15.107:0/1592206555 learned_addr learned my addr 172.21.15.107:0/1592206555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c326f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 -- 172.21.15.107:0/1592206555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 msgr2=0x5593c327fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 -- 172.21.15.107:0/1592206555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 msgr2=0x5593c326f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c326f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 -- 172.21.15.107:0/1592206555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c2fac7d0 con 0x5593c326e640 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c326f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0b400cc8b0 tx=0x7f0b40097670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803a0f9bc6c76485 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b37fff640 1 -- 172.21.15.107:0/1592206555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4009ade0 con 0x5593c326e640 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b37fff640 1 -- 172.21.15.107:0/1592206555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4009b9a0 con 0x5593c326e640 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.817+0000 7f0b37fff640 1 -- 172.21.15.107:0/1592206555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4009ac00 con 0x5593c326e640 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- 172.21.15.107:0/1592206555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 msgr2=0x5593c32806d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0b400cc8b0 tx=0x7f0b40097670 comp rx=0 tx=0).stop 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- 172.21.15.107:0/1592206555 shutdown_connections 2024-04-24T23:46:58.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c326f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/1592206555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c32806d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- 172.21.15.107:0/1592206555 >> 172.21.15.107:0/1592206555 conn(0x5593c32658a0 msgr2=0x5593c3264b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- 172.21.15.107:0/1592206555 shutdown_connections 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- 172.21.15.107:0/1592206555 wait complete. 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 -- start start 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.818+0000 7f0b57bf4740 1 Processor -- start 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a97c0 con 0x5593c326e640 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a90f0 con 0x5593c32741a0 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f0b501a6330 con 0x5593c326f030 2024-04-24T23:46:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 10 client.0 _ll_register_callbacks cb 0x7f0b38009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44774/0 (socket says 172.21.15.107:44774) 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3608116876 learned_addr learned my addr 172.21.15.107:0/3608116876 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4ed9d640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3608116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 msgr2=0x5593c327db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 msgr2=0x5593c327eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 -- 172.21.15.107:0/3608116876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c327f190 con 0x5593c326e640 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b4e59c640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0b400cc590 tx=0x7f0b400993c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc772b8175b8a671 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:58.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4009ade0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b400bb0b0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.819+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b4009b9a0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b4009b2a0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 10 client.16116 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5593c32eaec0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4009b9a0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0b400b8ce0 con 0x5593c326e640 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 10 client.16116 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 20 client.16116 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c32eb860 0x5593c32edc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 4 client.16116 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593c31d5f00 con 0x5593c32eb860 2024-04-24T23:46:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.820+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c32eb860 0x5593c32edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b35ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b45d9b640 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c32eb860 0x5593c32edc80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eba6c11f730ab576 server_cookie=30ca8be240220c52 in_seq=0 out_seq=0 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b35ffb640 1 -- 172.21.15.107:0/3608116876 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593c31d5f00 con 0x5593c32eb860 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b35ffb640 10 client.16116 handle_command_reply: tid=0 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 client.16116 shutdown 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 msgr2=0x5593c327d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0b400cc590 tx=0x7f0b400993c0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 shutdown_connections 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5593c32eb860 0x5593c32edc80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5593c32741a0 0x5593c327eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b35ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5593c326f030 0x5593c327db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 --2- 172.21.15.107:0/3608116876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5593c326e640 0x5593c327d650 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 >> 172.21.15.107:0/3608116876 conn(0x5593c32658a0 msgr2=0x5593c3268580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 shutdown_connections 2024-04-24T23:46:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 1 -- 172.21.15.107:0/3608116876 wait complete. 2024-04-24T23:46:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.821+0000 7f0b57bf4740 20 client.16116 trim_cache size 0 max 16384 2024-04-24T23:46:58.828 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 msgr2=0x7f0b3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 0x7f0b3c0a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0b4804d000 tx=0x7f0b4804e000 comp rx=0 tx=0).stop 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 msgr2=0x7f0b50197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000d950 tx=0x7f0b4000de10 comp rx=0 tx=0).stop 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 shutdown_connections 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f0b3c0a2b90 0x7f0b3c0a5050 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f0b50107480 0x7f0b5019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b50104b50 0x7f0b50198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 --2- 172.21.15.107:0/3177153132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b50102220 0x7f0b50197ac0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:58.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 >> 172.21.15.107:0/3177153132 conn(0x7f0b500fbc30 msgr2=0x7f0b50102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:58.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 shutdown_connections 2024-04-24T23:46:58.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:58.822+0000 7f0b54ee1640 1 -- 172.21.15.107:0/3177153132 wait complete. 2024-04-24T23:46:58.902 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:46:58.902 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 readers = 7668383.50 kB/sec 2024-04-24T23:46:58.902 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 readers = 7656688.82 kB/sec 2024-04-24T23:46:58.902 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 7668383.50 kB/sec 2024-04-24T23:46:58.903 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 7668383.50 kB/sec 2024-04-24T23:46:58.903 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 7668383.50 kB/sec 2024-04-24T23:46:58.903 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:46:59.463 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:46:59 smithi195 ceph-mon[33019]: pgmap v418: 129 pgs: 129 active+clean; 1.0 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 114 MiB/s wr, 61 op/s 2024-04-24T23:46:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:46:59 smithi170 ceph-mon[32714]: pgmap v418: 129 pgs: 129 active+clean; 1.0 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 114 MiB/s wr, 61 op/s 2024-04-24T23:46:59.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:46:59 smithi107 ceph-mon[29118]: pgmap v418: 129 pgs: 129 active+clean; 1.0 GiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 114 MiB/s wr, 61 op/s 2024-04-24T23:46:59.836 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T23:46:59.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:46:59.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T23:46:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3863692094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0103340 msgr2=0x7efdb0109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3863692094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0103340 0x7efdb0109bd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efda8009ca0 tx=0x7efda802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3863692094 shutdown_connections 2024-04-24T23:46:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3863692094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0103340 0x7efdb0109bd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3863692094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0102980 0x7efdb0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3863692094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0101790 0x7efdb0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3863692094 >> 172.21.15.107:0/3863692094 conn(0x7efdb00fcd20 msgr2=0x7efdb00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3863692094 shutdown_connections 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.951+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3863692094 wait complete. 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 Processor -- start 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 -- start start 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 0x7efdb0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0103340 0x7efdb019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010d440 con 0x7efdb0101790 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010d2c0 con 0x7efdb0102980 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdb5bc8640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010d5c0 con 0x7efdb0103340 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44794/0 (socket says 172.21.15.107:44794) 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 -- 172.21.15.107:0/3230738891 learned_addr learned my addr 172.21.15.107:0/3230738891 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaffff640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0103340 0x7efdb019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 0x7efdb0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 -- 172.21.15.107:0/3230738891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0103340 msgr2=0x7efdb019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0103340 0x7efdb019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 -- 172.21.15.107:0/3230738891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 msgr2=0x7efdb0197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 0x7efdb0197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaf7fe640 1 -- 172.21.15.107:0/3230738891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb019f690 con 0x7efdb0101790 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.952+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 0x7efdb0197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9c0 tx=0x7efda000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdacff9640 1 -- 172.21.15.107:0/3230738891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda000cd80 con 0x7efdb0101790 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb019f980 con 0x7efdb0101790 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdacff9640 1 -- 172.21.15.107:0/3230738891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda00035f0 con 0x7efdb0101790 2024-04-24T23:46:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdacff9640 1 -- 172.21.15.107:0/3230738891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0010690 con 0x7efdb0101790 2024-04-24T23:46:59.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.953+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb019ff20 con 0x7efdb0101790 2024-04-24T23:46:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.955+0000 7efdacff9640 1 -- 172.21.15.107:0/3230738891 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7efda00037a0 con 0x7efdb0101790 2024-04-24T23:46:59.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.955+0000 7efdb89b5740 1 Processor -- start 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 -- start start 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 0x563288d3c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d3ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdacff9640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 0x7efd980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563288c9a2c0 con 0x563288d36d90 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563288c42910 con 0x563288d3c8f0 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdb89b5740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x563288c96280 con 0x563288d37780 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44798/0 (socket says 172.21.15.107:44798) 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 0x563288d3c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 -- 172.21.15.107:0/525874413 learned_addr learned my addr 172.21.15.107:0/525874413 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaeffd640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d3ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdacff9640 1 -- 172.21.15.107:0/3230738891 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efda00cf9c0 con 0x7efdb0101790 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 0x7efd980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 -- 172.21.15.107:0/525874413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 msgr2=0x563288d3c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 0x563288d3c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 -- 172.21.15.107:0/525874413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 msgr2=0x563288d3ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d3ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaffff640 1 -- 172.21.15.107:0/525874413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563288c53fb0 con 0x563288d36d90 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaeffd640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d3ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.956+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 0x563288d3c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdaffff640 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7efda802f540 tx=0x7efda8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c990c231573a9468 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdaffff640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 0x7efd980a5050 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efda8031c10 tx=0x7efda8043000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efd95ffb640 1 -- 172.21.15.107:0/525874413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda8009820 con 0x563288d36d90 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efd95ffb640 1 -- 172.21.15.107:0/525874413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda8050070 con 0x563288d36d90 2024-04-24T23:46:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efd95ffb640 1 -- 172.21.15.107:0/525874413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda802fbd0 con 0x563288d36d90 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 -- 172.21.15.107:0/525874413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 msgr2=0x563288d371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7efda802f540 tx=0x7efda8031700 comp rx=0 tx=0).stop 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 -- 172.21.15.107:0/525874413 shutdown_connections 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d3ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d37780 0x563288d3c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 --2- 172.21.15.107:0/525874413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d36d90 0x563288d371b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 -- 172.21.15.107:0/525874413 >> 172.21.15.107:0/525874413 conn(0x563288cfe950 msgr2=0x563288d32810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 -- 172.21.15.107:0/525874413 shutdown_connections 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.957+0000 7efdb89b5740 1 -- 172.21.15.107:0/525874413 wait complete. 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 -- start start 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 Processor -- start 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d48730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563288d3f840 con 0x563288d37780 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563288c55e90 con 0x563288d36d90 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x563288d3f9c0 con 0x563288d3c8f0 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdb89b5740 10 client.0 _ll_register_callbacks cb 0x563288c91970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44810/0 (socket says 172.21.15.107:44810) 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdaeffd640 1 -- 172.21.15.107:0/2152150150 learned_addr learned my addr 172.21.15.107:0/2152150150 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.958+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaffff640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d48730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 -- 172.21.15.107:0/2152150150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 msgr2=0x563288d48730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d48730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 -- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 msgr2=0x563288d411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 -- 172.21.15.107:0/2152150150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563288a767e0 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdaeffd640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efda40107f0 tx=0x7efda4010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca123c73be3cd30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda400ed40 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efda4005020 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efda8050070 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efda4023050 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdb89b5740 10 client.16128 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563288ad6f70 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda8050070 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efda4018430 con 0x563288d37780 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdb89b5740 10 client.16128 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.959+0000 7efdb89b5740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdb89b5740 20 client.16128 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288d4cd30 0x563288d4f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdb89b5740 4 client.16128 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:46:59.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563288c6e5a0 con 0x563288d4cd30 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288d4cd30 0x563288d4f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288d4cd30 0x563288d4f1d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd9b067fc360b8f server_cookie=262bda37f95866a4 in_seq=0 out_seq=0 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.960+0000 7efd8effd640 10 client.16128 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efd8effd640 1 -- 172.21.15.107:0/2152150150 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563288c6e5a0 con 0x563288d4cd30 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efd8effd640 10 client.16128 handle_command_reply: tid=0 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 client.16128 shutdown 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 msgr2=0x563288d416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efda40107f0 tx=0x7efda4010cb0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 shutdown_connections 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288d4cd30 0x563288d4f1d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d48730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efd8effd640 0 client.16128 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d416f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 --2- 172.21.15.107:0/2152150150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 >> 172.21.15.107:0/2152150150 conn(0x563288cfe950 msgr2=0x563288d31950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 shutdown_connections 2024-04-24T23:46:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 1 -- 172.21.15.107:0/2152150150 wait complete. 2024-04-24T23:46:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.961+0000 7efdb89b5740 20 client.16128 trim_cache size 0 max 16384 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 Processor -- start 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 -- start start 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d35be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d486f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01aa4f0 con 0x563288d37780 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01ac030 con 0x563288d36d90 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdb89b5740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01aabc0 con 0x563288d3c8f0 2024-04-24T23:46:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44812/0 (socket says 172.21.15.107:44812) 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 -- 172.21.15.107:0/3120174085 learned_addr learned my addr 172.21.15.107:0/3120174085 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d35be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 -- 172.21.15.107:0/3120174085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 msgr2=0x563288d486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 -- 172.21.15.107:0/3120174085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 msgr2=0x563288d35be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d35be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 -- 172.21.15.107:0/3120174085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328896e170 con 0x563288d37780 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d35be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efda400b510 tx=0x7efda400e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13510c900edba717 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efd957fa640 1 -- 172.21.15.107:0/3120174085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda40056f0 con 0x563288d37780 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efd957fa640 1 -- 172.21.15.107:0/3120174085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda4018430 con 0x563288d37780 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.979+0000 7efd957fa640 1 -- 172.21.15.107:0/3120174085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda400f710 con 0x563288d37780 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- 172.21.15.107:0/3120174085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 msgr2=0x563288d36120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efda400b510 tx=0x7efda400e780 comp rx=0 tx=0).stop 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- 172.21.15.107:0/3120174085 shutdown_connections 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d3c8f0 0x563288d486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d36120 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3120174085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d36d90 0x563288d35be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- 172.21.15.107:0/3120174085 >> 172.21.15.107:0/3120174085 conn(0x563288cfe950 msgr2=0x563288d354d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- 172.21.15.107:0/3120174085 shutdown_connections 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- 172.21.15.107:0/3120174085 wait complete. 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.980+0000 7efdb89b5740 1 -- start start 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 Processor -- start 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d36d90 0x563288d46640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d47b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01ab960 con 0x563288d37780 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01ab290 con 0x563288d3c8f0 2024-04-24T23:46:59.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7efdb01a84d0 con 0x563288d36d90 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdb89b5740 10 client.0 _ll_register_callbacks cb 0x7efda400b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44814/0 (socket says 172.21.15.107:44814) 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 -- 172.21.15.107:0/3723208641 learned_addr learned my addr 172.21.15.107:0/3723208641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d36d90 0x563288d46640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaffff640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d47b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 -- 172.21.15.107:0/3723208641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d36d90 msgr2=0x563288d46640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d36d90 0x563288d46640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 -- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 msgr2=0x563288d47b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d47b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 -- 172.21.15.107:0/3723208641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563288d48180 con 0x563288d37780 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efdaeffd640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efda40189c0 tx=0x7efda400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a2a170e1a142ad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.981+0000 7efd8effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:46:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4018a90 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efda4018c30 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efda4018430 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efda4045050 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 10 client.16134 fetch_fsmap learned FSMap version 29 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563288db3ab0 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda4018430 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efda40315d0 con 0x563288d37780 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 10 client.16134 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 20 client.16134 populate_metadata read hostname 'smithi107' 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288db4420 0x563288db6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 4 client.16134 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563288ca7800 con 0x563288db4420 2024-04-24T23:46:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.982+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288db4420 0x563288db6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:46:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.983+0000 7efd8effd640 10 client.16134 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.983+0000 7efdaf7fe640 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288db4420 0x563288db6840 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3d3d80d42f9a9e server_cookie=8d60f034f8f2eae2 in_seq=0 out_seq=0 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efd8effd640 1 -- 172.21.15.107:0/3723208641 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563288ca7800 con 0x563288db4420 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efd8effd640 10 client.16134 handle_command_reply: tid=0 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 client.16134 shutdown 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 msgr2=0x563288d46b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efda40189c0 tx=0x7efda400bd60 comp rx=0 tx=0).stop 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 shutdown_connections 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563288db4420 0x563288db6840 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563288d3c8f0 0x563288d47b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efd8effd640 0 client.16134 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:46:59.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563288d37780 0x563288d46b80 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 --2- 172.21.15.107:0/3723208641 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563288d36d90 0x563288d46640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 >> 172.21.15.107:0/3723208641 conn(0x563288cfe950 msgr2=0x563288d35d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 shutdown_connections 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 1 -- 172.21.15.107:0/3723208641 wait complete. 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.984+0000 7efdb89b5740 20 client.16134 trim_cache size 0 max 16384 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "74f19cc6-533b-4f46-a85b-69fcd9bd8547", 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-04-24T23:46:59.990 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:46:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 msgr2=0x7efd980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 0x7efd980a5050 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efda8031c10 tx=0x7efda8043000 comp rx=0 tx=0).stop 2024-04-24T23:46:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 msgr2=0x7efdb0197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:46:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9c0 tx=0x7efda000ee80 comp rx=0 tx=0).stop 2024-04-24T23:46:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 shutdown_connections 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7efd980a2b90 0x7efd980a5050 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7efdb0103340 0x7efdb019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb0102980 0x7efdb0197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 --2- 172.21.15.107:0/3230738891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdb0101790 0x7efdb0197a70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 >> 172.21.15.107:0/3230738891 conn(0x7efdb00fcd20 msgr2=0x7efdb0105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 shutdown_connections 2024-04-24T23:46:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:46:59.985+0000 7efdb5bc8640 1 -- 172.21.15.107:0/3230738891 wait complete. 2024-04-24T23:46:59.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:46:59.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- 172.21.15.107:0/2008694330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 msgr2=0x7f6f04102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/2008694330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f04102040 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009900 tx=0x7f6f0002a410 comp rx=0 tx=0).stop 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- 172.21.15.107:0/2008694330 shutdown_connections 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/2008694330 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f04108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/2008694330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 0x7f6f04100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/2008694330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f04102040 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- 172.21.15.107:0/2008694330 >> 172.21.15.107:0/2008694330 conn(0x7f6f040fb380 msgr2=0x7f6f040fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- 172.21.15.107:0/2008694330 shutdown_connections 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- 172.21.15.107:0/2008694330 wait complete. 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 Processor -- start 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.096+0000 7f6f0b42a640 1 -- start start 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 0x7f6f04199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f0419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0410d440 con 0x7f6f04101c20 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0410d2c0 con 0x7f6f04100950 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0410d5c0 con 0x7f6f04101310 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44830/0 (socket says 172.21.15.107:44830) 2024-04-24T23:47:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 -- 172.21.15.107:0/298819298 learned_addr learned my addr 172.21.15.107:0/298819298 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f089c5640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f0419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f091c6640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 0x7f6f04199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 -- 172.21.15.107:0/298819298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 msgr2=0x7f6f0419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f0419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 -- 172.21.15.107:0/298819298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 msgr2=0x7f6f04199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 0x7f6f04199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 -- 172.21.15.107:0/298819298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f041a1800 con 0x7f6f04101c20 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f089c5640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f0419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f099c7640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00cce0 tx=0x7f6efc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6efa7fc640 1 -- 172.21.15.107:0/298819298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00f450 con 0x7f6f04101c20 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6efa7fc640 1 -- 172.21.15.107:0/298819298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6efc00fad0 con 0x7f6f04101c20 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f041a1af0 con 0x7f6f04101c20 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6efa7fc640 1 -- 172.21.15.107:0/298819298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc018c10 con 0x7f6f04101c20 2024-04-24T23:47:00.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.097+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f041a2040 con 0x7f6f04101c20 2024-04-24T23:47:00.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.099+0000 7f6efa7fc640 1 -- 172.21.15.107:0/298819298 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6efc00f5f0 con 0x7f6f04101c20 2024-04-24T23:47:00.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 Processor -- start 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 -- start start 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52604730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb52604da0 0x55eb52607230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eb525636b0 con 0x55eb525ff240 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55eb5250b880 con 0x55eb52604da0 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f0e215740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55eb5255f670 con 0x55eb525ffc30 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6efa7fc640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 0x7f6ef00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44842/0 (socket says 172.21.15.107:44842) 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 -- 172.21.15.107:0/3858978137 learned_addr learned my addr 172.21.15.107:0/3858978137 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f099c7640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 0x7f6ef00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6efa7fc640 1 -- 172.21.15.107:0/298819298 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6efc0cfb00 con 0x7f6f04101c20 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f091c6640 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52604730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb52604da0 0x55eb52607230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 -- 172.21.15.107:0/3858978137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 msgr2=0x55eb52604730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52604730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 -- 172.21.15.107:0/3858978137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb52604da0 msgr2=0x55eb52607230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb52604da0 0x55eb52607230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.100+0000 7f6f089c5640 1 -- 172.21.15.107:0/3858978137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb523417d0 con 0x55eb525ff240 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.101+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40145f0 tx=0x7f6ef400b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc0e57e7b32d2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.101+0000 7f6f099c7640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 0x7f6ef00a50d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0983f0 tx=0x7f6efc06f050 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.101+0000 7f6eef7fe640 1 -- 172.21.15.107:0/3858978137 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4005220 con 0x55eb525ff240 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.101+0000 7f6eef7fe640 1 -- 172.21.15.107:0/3858978137 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4005e80 con 0x55eb525ff240 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.101+0000 7f6eef7fe640 1 -- 172.21.15.107:0/3858978137 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4012710 con 0x55eb525ff240 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- 172.21.15.107:0/3858978137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 msgr2=0x55eb525ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40145f0 tx=0x7f6ef400b5d0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- 172.21.15.107:0/3858978137 shutdown_connections 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb52604da0 0x55eb52607230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52604730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3858978137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ff240 0x55eb525ff660 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- 172.21.15.107:0/3858978137 >> 172.21.15.107:0/3858978137 conn(0x55eb525fa8b0 msgr2=0x55eb525facb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- 172.21.15.107:0/3858978137 shutdown_connections 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- 172.21.15.107:0/3858978137 wait complete. 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- start start 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 Processor -- start 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb526095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb52610b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eb52607d50 con 0x55eb525ffc30 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55eb5251ef50 con 0x55eb525ff240 2024-04-24T23:47:00.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55eb52607ed0 con 0x55eb52604da0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.102+0000 7f6f0e215740 10 client.0 _ll_register_callbacks cb 0x55eb5255a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44844/0 (socket says 172.21.15.107:44844) 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 -- 172.21.15.107:0/3337358856 learned_addr learned my addr 172.21.15.107:0/3337358856 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb526095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb52610b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 -- 172.21.15.107:0/3337358856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 msgr2=0x55eb52610b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb52610b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 -- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 msgr2=0x55eb526095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb526095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 -- 172.21.15.107:0/3337358856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb522773f0 con 0x55eb525ffc30 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb526095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f091c6640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009900 tx=0x7f6f00031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941818fda3c3553a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6eecff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0003f070 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6f00004450 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb52570bf0 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6f00035070 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f0e215740 10 client.16146 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eb522370f0 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.103+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eb52570bf0 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6f000035f0 con 0x55eb525ffc30 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 10 client.16146 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 20 client.16146 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb52615420 0x55eb526178c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 4 client.16146 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb5256af40 con 0x55eb52615420 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb52615420 0x55eb526178c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6eecff9640 10 client.16146 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.104+0000 7f6f089c5640 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb52615420 0x55eb526178c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bff1da983eda40c server_cookie=9f154996e0d1f95f in_seq=0 out_seq=0 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6eecff9640 1 -- 172.21.15.107:0/3337358856 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb5256af40 con 0x55eb52615420 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6eecff9640 10 client.16146 handle_command_reply: tid=0 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 client.16146 shutdown 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 msgr2=0x55eb52609b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009900 tx=0x7f6f00031960 comp rx=0 tx=0).stop 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 shutdown_connections 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb52615420 0x55eb526178c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb52610b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6eecff9640 0 client.16146 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb52609b20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 --2- 172.21.15.107:0/3337358856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb526095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 >> 172.21.15.107:0/3337358856 conn(0x55eb525fa8b0 msgr2=0x55eb525f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 shutdown_connections 2024-04-24T23:47:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.105+0000 7f6f0e215740 1 -- 172.21.15.107:0/3337358856 wait complete. 2024-04-24T23:47:00.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.106+0000 7f6f0e215740 20 client.16146 trim_cache size 0 max 16384 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.122+0000 7f6f0e215740 1 Processor -- start 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.122+0000 7f6f0e215740 1 -- start start 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb525fd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52610b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ac5e0 con 0x55eb52604da0 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ae120 con 0x55eb525ff240 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f0e215740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041accb0 con 0x55eb525ffc30 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44848/0 (socket says 172.21.15.107:44848) 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 -- 172.21.15.107:0/1507575555 learned_addr learned my addr 172.21.15.107:0/1507575555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb525fd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 -- 172.21.15.107:0/1507575555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 msgr2=0x55eb52610b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f091c6640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52610b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52610b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 -- 172.21.15.107:0/1507575555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 msgr2=0x55eb525fd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb525fd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 -- 172.21.15.107:0/1507575555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb525ec090 con 0x55eb52604da0 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f091c6640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52610b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6f089c5640 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40145f0 tx=0x7f6ef400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33c194f70557276 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6eeeffd640 1 -- 172.21.15.107:0/1507575555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4005320 con 0x55eb52604da0 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6eeeffd640 1 -- 172.21.15.107:0/1507575555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef40054c0 con 0x55eb52604da0 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.123+0000 7f6eeeffd640 1 -- 172.21.15.107:0/1507575555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40049f0 con 0x55eb52604da0 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- 172.21.15.107:0/1507575555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 msgr2=0x55eb525fdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40145f0 tx=0x7f6ef400b3b0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- 172.21.15.107:0/1507575555 shutdown_connections 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb52604da0 0x55eb525fdb70 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb525ffc30 0x55eb52610b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 --2- 172.21.15.107:0/1507575555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb525fd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- 172.21.15.107:0/1507575555 >> 172.21.15.107:0/1507575555 conn(0x55eb525fa8b0 msgr2=0x55eb52600ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- 172.21.15.107:0/1507575555 shutdown_connections 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- 172.21.15.107:0/1507575555 wait complete. 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 -- start start 2024-04-24T23:47:00.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.124+0000 7f6f0e215740 1 Processor -- start 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb5260e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb5260f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ada50 con 0x55eb525ffc30 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041ad380 con 0x55eb525ff240 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041aa5c0 con 0x55eb52604da0 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 10 client.0 _ll_register_callbacks cb 0x7f6f00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44850/0 (socket says 172.21.15.107:44850) 2024-04-24T23:47:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 -- 172.21.15.107:0/935227203 learned_addr learned my addr 172.21.15.107:0/935227203 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f089c5640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb5260e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f089c5640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb5260f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 -- 172.21.15.107:0/935227203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 msgr2=0x55eb5260f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb5260f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 -- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 msgr2=0x55eb5260e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb5260e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 -- 172.21.15.107:0/935227203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb5260fc90 con 0x55eb525ffc30 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f089c5640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb5260e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f091c6640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6f000314e0 tx=0x7f6f0002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9196818e1a126d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6eecff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:00.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0003f030 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6f000319d0 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef40054c0 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6f00031bd0 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 10 client.16152 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.125+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eb5267ba00 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef40054c0 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6f00004960 con 0x55eb525ffc30 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 10 client.16152 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 20 client.16152 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb5267c240 0x55eb5267e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 4 client.16152 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:47:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eb524e86f0 con 0x55eb5267c240 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f089c5640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb5267c240 0x55eb5267e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6eecff9640 10 client.16152 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.126+0000 7f6f089c5640 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb5267c240 0x55eb5267e660 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5679859f2e829de8 server_cookie=ac24f067a9b5322f in_seq=0 out_seq=0 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6eecff9640 1 -- 172.21.15.107:0/935227203 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eb524e86f0 con 0x55eb5267c240 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6eecff9640 10 client.16152 handle_command_reply: tid=0 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 client.16152 shutdown 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 msgr2=0x55eb5260e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6f000314e0 tx=0x7f6f0002a410 comp rx=0 tx=0).stop 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 shutdown_connections 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55eb5267c240 0x55eb5267e660 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55eb52604da0 0x55eb5260f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6eecff9640 0 client.16152 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb525ffc30 0x55eb5260e690 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 --2- 172.21.15.107:0/935227203 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55eb525ff240 0x55eb5260e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 >> 172.21.15.107:0/935227203 conn(0x55eb525fa8b0 msgr2=0x55eb525f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 shutdown_connections 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 1 -- 172.21.15.107:0/935227203 wait complete. 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.127+0000 7f6f0e215740 20 client.16152 trim_cache size 0 max 16384 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "74f19cc6-533b-4f46-a85b-69fcd9bd8547": { 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "74f19cc6-533b-4f46-a85b-69fcd9bd8547", 2024-04-24T23:47:00.136 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-04-24T23:47:00.137 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:47:00.137 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:47:00.137 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:47:00.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 msgr2=0x7f6ef00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 0x7f6ef00a50d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0983f0 tx=0x7f6efc06f050 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 msgr2=0x7f6f041a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00cce0 tx=0x7f6efc00a620 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 shutdown_connections 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f6ef00a2c10 0x7f6ef00a50d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f04101c20 0x7f6f041a1190 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6f04101310 0x7f6f0419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 --2- 172.21.15.107:0/298819298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f04100950 0x7f6f04199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 >> 172.21.15.107:0/298819298 conn(0x7f6f040fb380 msgr2=0x7f6f04109470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 shutdown_connections 2024-04-24T23:47:00.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:00.128+0000 7f6f0b42a640 1 -- 172.21.15.107:0/298819298 wait complete. 2024-04-24T23:47:00.140 INFO:tasks.cephfs.filesystem:scrub status for tag:74f19cc6-533b-4f46-a85b-69fcd9bd8547 - {'path': '/', 'tag': '74f19cc6-533b-4f46-a85b-69fcd9bd8547', 'options': 'recursive,force'} 2024-04-24T23:47:01.151 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:01.151 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 re-readers = 6940125.50 kB/sec 2024-04-24T23:47:01.151 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 re-readers = 6930174.07 kB/sec 2024-04-24T23:47:01.151 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 6940125.50 kB/sec 2024-04-24T23:47:01.152 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 6940125.50 kB/sec 2024-04-24T23:47:01.152 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 6940125.50 kB/sec 2024-04-24T23:47:01.152 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 1048576.00 kB 2024-04-24T23:47:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:01 smithi195 ceph-mon[33019]: scrub queued for path: / 2024-04-24T23:47:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:01 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:01 smithi195 ceph-mon[33019]: scrub summary: active paths [/] 2024-04-24T23:47:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:01 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:01 smithi195 ceph-mon[33019]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 171 MiB/s wr, 77 op/s 2024-04-24T23:47:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:01 smithi170 ceph-mon[32714]: scrub queued for path: / 2024-04-24T23:47:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:01 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:01 smithi170 ceph-mon[32714]: scrub summary: active paths [/] 2024-04-24T23:47:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:01 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:01 smithi170 ceph-mon[32714]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 171 MiB/s wr, 77 op/s 2024-04-24T23:47:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:01 smithi107 ceph-mon[29118]: scrub queued for path: / 2024-04-24T23:47:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:01 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:01 smithi107 ceph-mon[29118]: scrub summary: active paths [/] 2024-04-24T23:47:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:01 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:01 smithi107 ceph-mon[29118]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 171 MiB/s wr, 77 op/s 2024-04-24T23:47:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:03 smithi107 ceph-mon[29118]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 78 op/s 2024-04-24T23:47:03.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:03 smithi195 ceph-mon[33019]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 78 op/s 2024-04-24T23:47:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:03 smithi170 ceph-mon[32714]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 78 op/s 2024-04-24T23:47:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:05 smithi195 ceph-mon[33019]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 MiB/s wr, 75 op/s 2024-04-24T23:47:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:05 smithi195 ceph-mon[33019]: scrub summary: idle 2024-04-24T23:47:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:05 smithi170 ceph-mon[32714]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 MiB/s wr, 75 op/s 2024-04-24T23:47:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:05 smithi170 ceph-mon[32714]: scrub summary: idle 2024-04-24T23:47:05.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:05 smithi107 ceph-mon[29118]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 170 MiB/s wr, 75 op/s 2024-04-24T23:47:05.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:05 smithi107 ceph-mon[29118]: scrub summary: idle 2024-04-24T23:47:07.523 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.524 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.524 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.524 INFO:tasks.workunit.client.0.smithi107.stdout:iozone test complete. 2024-04-24T23:47:07.525 INFO:tasks.workunit.client.0.smithi107.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-24T23:47:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:07 smithi195 ceph-mon[33019]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 143 MiB/s wr, 69 op/s 2024-04-24T23:47:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:07 smithi170 ceph-mon[32714]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 143 MiB/s wr, 69 op/s 2024-04-24T23:47:07.589 INFO:tasks.workunit.client.0.smithi107.stdout: Iozone: Performance Test of File I/O 2024-04-24T23:47:07.589 INFO:tasks.workunit.client.0.smithi107.stdout: Version $Revision: 3.494 $ 2024-04-24T23:47:07.589 INFO:tasks.workunit.client.0.smithi107.stdout: Compiled for 64 bit mode. 2024-04-24T23:47:07.589 INFO:tasks.workunit.client.0.smithi107.stdout: Build: linux-AMD64 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-24T23:47:07.590 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Run began: Wed Apr 24 23:47:07 2024 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Include close in write timing 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Include fsync in write timing 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: File size set to 10485760 kB 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Record Size 1024 kB 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Output is in kBytes/sec 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Time Resolution = 0.000001 seconds. 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache size set to 1024 kBytes. 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Processor cache line size set to 32 bytes. 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: File stride size set to 17 * record size. 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Throughput test with 1 process 2024-04-24T23:47:07.591 INFO:tasks.workunit.client.0.smithi107.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-04-24T23:47:07.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:07 smithi107 ceph-mon[29118]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 143 MiB/s wr, 69 op/s 2024-04-24T23:47:09.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:09 smithi195 ceph-mon[33019]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 96 MiB/s wr, 42 op/s 2024-04-24T23:47:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:09 smithi170 ceph-mon[32714]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 96 MiB/s wr, 42 op/s 2024-04-24T23:47:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:09 smithi107 ceph-mon[29118]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 96 MiB/s wr, 42 op/s 2024-04-24T23:47:11.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:11 smithi195 ceph-mon[33019]: pgmap v424: 129 pgs: 129 active+clean; 1.4 GiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 89 MiB/s wr, 33 op/s 2024-04-24T23:47:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:11 smithi170 ceph-mon[32714]: pgmap v424: 129 pgs: 129 active+clean; 1.4 GiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 89 MiB/s wr, 33 op/s 2024-04-24T23:47:11.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:11 smithi107 ceph-mon[29118]: pgmap v424: 129 pgs: 129 active+clean; 1.4 GiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 89 MiB/s wr, 33 op/s 2024-04-24T23:47:13.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:13 smithi195 ceph-mon[33019]: pgmap v425: 129 pgs: 129 active+clean; 1.6 GiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 64 MiB/s wr, 140 op/s 2024-04-24T23:47:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:13 smithi170 ceph-mon[32714]: pgmap v425: 129 pgs: 129 active+clean; 1.6 GiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 64 MiB/s wr, 140 op/s 2024-04-24T23:47:13.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:13 smithi107 ceph-mon[29118]: pgmap v425: 129 pgs: 129 active+clean; 1.6 GiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 64 MiB/s wr, 140 op/s 2024-04-24T23:47:15.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:15 smithi195 ceph-mon[33019]: pgmap v426: 129 pgs: 129 active+clean; 2.8 GiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 187 MiB/s wr, 187 op/s 2024-04-24T23:47:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:15 smithi170 ceph-mon[32714]: pgmap v426: 129 pgs: 129 active+clean; 2.8 GiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 187 MiB/s wr, 187 op/s 2024-04-24T23:47:15.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:15 smithi107 ceph-mon[29118]: pgmap v426: 129 pgs: 129 active+clean; 2.8 GiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 187 MiB/s wr, 187 op/s 2024-04-24T23:47:17.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:17 smithi195 ceph-mon[33019]: pgmap v427: 129 pgs: 129 active+clean; 4.7 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 404 MiB/s wr, 304 op/s 2024-04-24T23:47:17.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:17 smithi170 ceph-mon[32714]: pgmap v427: 129 pgs: 129 active+clean; 4.7 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 404 MiB/s wr, 304 op/s 2024-04-24T23:47:17.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:17 smithi107 ceph-mon[29118]: pgmap v427: 129 pgs: 129 active+clean; 4.7 GiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 404 MiB/s wr, 304 op/s 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 initial writers = 947652.31 kB/sec 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 initial writers = 947630.00 kB/sec 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 947652.31 kB/sec 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 947652.31 kB/sec 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 947652.31 kB/sec 2024-04-24T23:47:18.687 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 10485760.00 kB 2024-04-24T23:47:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:19 smithi195 ceph-mon[33019]: pgmap v428: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 561 MiB/s wr, 388 op/s 2024-04-24T23:47:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:19 smithi170 ceph-mon[32714]: pgmap v428: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 561 MiB/s wr, 388 op/s 2024-04-24T23:47:19.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:19 smithi107 ceph-mon[29118]: pgmap v428: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 561 MiB/s wr, 388 op/s 2024-04-24T23:47:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:21 smithi195 ceph-mon[33019]: pgmap v429: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 835 MiB/s wr, 457 op/s 2024-04-24T23:47:21.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:21 smithi170 ceph-mon[32714]: pgmap v429: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 835 MiB/s wr, 457 op/s 2024-04-24T23:47:21.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:21 smithi107 ceph-mon[29118]: pgmap v429: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 835 MiB/s wr, 457 op/s 2024-04-24T23:47:23.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:23 smithi195 ceph-mon[33019]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 821 MiB/s wr, 454 op/s 2024-04-24T23:47:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:23 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:47:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:23 smithi170 ceph-mon[32714]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 821 MiB/s wr, 454 op/s 2024-04-24T23:47:23.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:23 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:47:23.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:23 smithi107 ceph-mon[29118]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 821 MiB/s wr, 454 op/s 2024-04-24T23:47:23.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:23 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:47:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:25 smithi195 ceph-mon[33019]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 811 MiB/s wr, 362 op/s 2024-04-24T23:47:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:47:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:47:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:47:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:25 smithi170 ceph-mon[32714]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 811 MiB/s wr, 362 op/s 2024-04-24T23:47:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:47:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:47:25.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:47:25.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:25 smithi107 ceph-mon[29118]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 811 MiB/s wr, 362 op/s 2024-04-24T23:47:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:47:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:47:25.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:47:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:27 smithi195 ceph-mon[33019]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 764 MiB/s wr, 334 op/s 2024-04-24T23:47:27.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:27 smithi170 ceph-mon[32714]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 764 MiB/s wr, 334 op/s 2024-04-24T23:47:27.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:27 smithi107 ceph-mon[29118]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 764 MiB/s wr, 334 op/s 2024-04-24T23:47:29.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:29 smithi195 ceph-mon[33019]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 663 MiB/s wr, 330 op/s 2024-04-24T23:47:29.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:29 smithi170 ceph-mon[32714]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 663 MiB/s wr, 330 op/s 2024-04-24T23:47:29.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:29 smithi107 ceph-mon[29118]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 663 MiB/s wr, 330 op/s 2024-04-24T23:47:30.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:47:30.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:47:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.473+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1144459500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4105330 msgr2=0x7fc4d41057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.473+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1144459500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4105330 0x7fc4d41057b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009ca0 tx=0x7fc4cc02a8f0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.476+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1144459500 shutdown_connections 2024-04-24T23:47:30.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.476+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1144459500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4105330 0x7fc4d41057b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.476+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1144459500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4102980 0x7fc4d4104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.476+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1144459500 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4d4100010 0x7fc4d4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.476+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1144459500 >> 172.21.15.107:0/1144459500 conn(0x7fc4d40fbc30 msgr2=0x7fc4d40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.482+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1144459500 shutdown_connections 2024-04-24T23:47:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.482+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1144459500 wait complete. 2024-04-24T23:47:30.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.483+0000 7fc4db4c3640 1 Processor -- start 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.486+0000 7fc4db4c3640 1 -- start start 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4102980 0x7fc4d410ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4d4105330 0x7fc4d410fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d4106c40 con 0x7fc4d4100010 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d4106ac0 con 0x7fc4d4102980 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4db4c3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d4106dc0 con 0x7fc4d4105330 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43348/0 (socket says 172.21.15.107:43348) 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.487+0000 7fc4d925f640 1 -- 172.21.15.107:0/1033071293 learned_addr learned my addr 172.21.15.107:0/1033071293 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.488+0000 7fc4d925f640 1 -- 172.21.15.107:0/1033071293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4d4105330 msgr2=0x7fc4d410fa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.488+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4d4105330 0x7fc4d410fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.488+0000 7fc4d925f640 1 -- 172.21.15.107:0/1033071293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4102980 msgr2=0x7fc4d410ea30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.488+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4102980 0x7fc4d410ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.488+0000 7fc4d925f640 1 -- 172.21.15.107:0/1033071293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d41100e0 con 0x7fc4d4100010 2024-04-24T23:47:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.489+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000b450 tx=0x7fc4d000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.490+0000 7fc4ca7fc640 1 -- 172.21.15.107:0/1033071293 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0015020 con 0x7fc4d4100010 2024-04-24T23:47:30.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.490+0000 7fc4ca7fc640 1 -- 172.21.15.107:0/1033071293 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d0011d70 con 0x7fc4d4100010 2024-04-24T23:47:30.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.490+0000 7fc4ca7fc640 1 -- 172.21.15.107:0/1033071293 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000f340 con 0x7fc4d4100010 2024-04-24T23:47:30.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.490+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d41103d0 con 0x7fc4d4100010 2024-04-24T23:47:30.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.490+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d4110920 con 0x7fc4d4100010 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.496+0000 7fc4ca7fc640 1 -- 172.21.15.107:0/1033071293 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc4d0018020 con 0x7fc4d4100010 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.497+0000 7fc4ca7fc640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 0x7fc4c00a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.497+0000 7fc4ca7fc640 1 -- 172.21.15.107:0/1033071293 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc4d00cfe40 con 0x7fc4d4100010 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 Processor -- start 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 -- start start 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fe1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fe1db0 0x5622d3fe4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5622d3f2f6b0 con 0x5622d3fe1db0 2024-04-24T23:47:30.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5622d3ed7880 con 0x5622d3fdcc40 2024-04-24T23:47:30.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.498+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5622d3f2b670 con 0x5622d3fdc250 2024-04-24T23:47:30.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.499+0000 7fc4d8a5e640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 0x7fc4c00a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.499+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.499+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.107:45472/0 (socket says 172.21.15.107:45472) 2024-04-24T23:47:30.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.499+0000 7fc4d925f640 1 -- 172.21.15.107:0/2046309717 learned_addr learned my addr 172.21.15.107:0/2046309717 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.500+0000 7fc4d925f640 1 -- 172.21.15.107:0/2046309717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 msgr2=0x5622d3fe1740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.500+0000 7fc4d925f640 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fe1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.500+0000 7fc4d925f640 1 -- 172.21.15.107:0/2046309717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fe1db0 msgr2=0x5622d3fe4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.500+0000 7fc4d925f640 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fe1db0 0x5622d3fe4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.501+0000 7fc4d925f640 1 -- 172.21.15.107:0/2046309717 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d3d0d7d0 con 0x5622d3fdc250 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.501+0000 7fc4d925f640 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0098ca0 tx=0x7fc4d009cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a047f720a845c27e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.502+0000 7fc4bf7fe640 1 -- 172.21.15.107:0/2046309717 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00a4f40 con 0x5622d3fdc250 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.502+0000 7fc4bf7fe640 1 -- 172.21.15.107:0/2046309717 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d00b5210 con 0x5622d3fdc250 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2046309717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 msgr2=0x5622d3fdc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0098ca0 tx=0x7fc4d009cb10 comp rx=0 tx=0).stop 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2046309717 shutdown_connections 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fe1db0 0x5622d3fe4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fe1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2046309717 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdc250 0x5622d3fdc670 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.503+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2046309717 >> 172.21.15.107:0/2046309717 conn(0x5622d3fc68b0 msgr2=0x5622d3fc6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.506+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2046309717 shutdown_connections 2024-04-24T23:47:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.506+0000 7fc4d8a5e640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 0x7fc4c00a51e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5622d3fdd040 tx=0x7fc4c4005b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.508+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2046309717 wait complete. 2024-04-24T23:47:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.508+0000 7fc4de1cb740 1 -- start start 2024-04-24T23:47:30.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 Processor -- start 2024-04-24T23:47:30.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fcca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5622d3fcb1a0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5622d3eeaf50 con 0x5622d3fdcc40 2024-04-24T23:47:30.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.513+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5622d3fcb320 con 0x5622d3fe1db0 2024-04-24T23:47:30.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4de1cb740 10 client.0 _ll_register_callbacks cb 0x5622d3f26000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:30.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43370/0 (socket says 172.21.15.107:43370) 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 -- 172.21.15.107:0/2067868712 learned_addr learned my addr 172.21.15.107:0/2067868712 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 -- 172.21.15.107:0/2067868712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 msgr2=0x5622d3fccfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fccfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 -- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 msgr2=0x5622d3fcca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.514+0000 7fc4d9a60640 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fcca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.515+0000 7fc4d9a60640 1 -- 172.21.15.107:0/2067868712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d3c433f0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.515+0000 7fc4d9a60640 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0316b0 tx=0x7fc4cc0316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e032ca9ccc64794 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.515+0000 7fc4bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc030a00 con 0x5622d3fd1fe0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5622d3f3cbf0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc4cc030ba0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4cc04d050 con 0x5622d3fd1fe0 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4de1cb740 10 client.16161 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:30.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.516+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622d3c030f0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.517+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5622d3f3cbf0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc4cc03b400 con 0x5622d3fd1fe0 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 10 client.16161 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 20 client.16161 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d3fd9240 0x5622d3fdb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 4 client.16161 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:30.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.518+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622d3f36f40 con 0x5622d3fd9240 2024-04-24T23:47:30.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.522+0000 7fc4d925f640 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d3fd9240 0x5622d3fdb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.541+0000 7fc4bcff9640 10 client.16161 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.541+0000 7fc4d925f640 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d3fd9240 0x5622d3fdb700 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb407c9a406a337d server_cookie=80e8ebbc73f7a81d in_seq=0 out_seq=0 2024-04-24T23:47:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.543+0000 7fc4bcff9640 1 -- 172.21.15.107:0/2067868712 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622d3f36f40 con 0x5622d3fd9240 2024-04-24T23:47:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.543+0000 7fc4bcff9640 10 client.16161 handle_command_reply: tid=0 2024-04-24T23:47:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.543+0000 7fc4de1cb740 1 client.16161 shutdown 2024-04-24T23:47:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.545+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 msgr2=0x5622d3fd4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.545+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0316b0 tx=0x7fc4cc0316e0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 shutdown_connections 2024-04-24T23:47:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d3fd9240 0x5622d3fdb700 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fd4470 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/2067868712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fcca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 >> 172.21.15.107:0/2067868712 conn(0x5622d3fc68b0 msgr2=0x5622d3fdc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.549+0000 7fc4bcff9640 0 client.16161 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:30.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.553+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 shutdown_connections 2024-04-24T23:47:30.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.553+0000 7fc4de1cb740 1 -- 172.21.15.107:0/2067868712 wait complete. 2024-04-24T23:47:30.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.553+0000 7fc4de1cb740 20 client.16161 trim_cache size 0 max 16384 2024-04-24T23:47:30.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.600+0000 7fc4de1cb740 1 Processor -- start 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.603+0000 7fc4de1cb740 1 -- start start 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fc9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fc9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b800fb30 con 0x5622d3fd1fe0 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b8011670 con 0x5622d3fdcc40 2024-04-24T23:47:30.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.604+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b8010200 con 0x5622d3fe1db0 2024-04-24T23:47:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.609+0000 7fc4d8a5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fc9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.609+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.609+0000 7fc4d9a60640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fc9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.609+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43374/0 (socket says 172.21.15.107:43374) 2024-04-24T23:47:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.609+0000 7fc4d925f640 1 -- 172.21.15.107:0/1786137637 learned_addr learned my addr 172.21.15.107:0/1786137637 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.613+0000 7fc4d925f640 1 -- 172.21.15.107:0/1786137637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 msgr2=0x5622d3fc9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.613+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fc9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.613+0000 7fc4d925f640 1 -- 172.21.15.107:0/1786137637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 msgr2=0x5622d3fc9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.613+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fc9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.613+0000 7fc4d925f640 1 -- 172.21.15.107:0/1786137637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d3fdd4d0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.617+0000 7fc4d925f640 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00cd8e0 tx=0x7fc4d00bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bbbfe529796313 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.618+0000 7fc4beffd640 1 -- 172.21.15.107:0/1786137637 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00b5210 con 0x5622d3fd1fe0 2024-04-24T23:47:30.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.618+0000 7fc4beffd640 1 -- 172.21.15.107:0/1786137637 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d00ba070 con 0x5622d3fd1fe0 2024-04-24T23:47:30.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.618+0000 7fc4beffd640 1 -- 172.21.15.107:0/1786137637 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00bc310 con 0x5622d3fd1fe0 2024-04-24T23:47:30.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.619+0000 7fc4de1cb740 1 -- 172.21.15.107:0/1786137637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 msgr2=0x5622d3fc6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.619+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00cd8e0 tx=0x7fc4d00bc040 comp rx=0 tx=0).stop 2024-04-24T23:47:30.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.620+0000 7fc4de1cb740 1 -- 172.21.15.107:0/1786137637 shutdown_connections 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.620+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fe1db0 0x5622d3fc9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.620+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fdcc40 0x5622d3fc9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.620+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/1786137637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d3fc6c90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.620+0000 7fc4de1cb740 1 -- 172.21.15.107:0/1786137637 >> 172.21.15.107:0/1786137637 conn(0x5622d3fc68b0 msgr2=0x5622d3fddf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.621+0000 7fc4de1cb740 1 -- 172.21.15.107:0/1786137637 shutdown_connections 2024-04-24T23:47:30.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.622+0000 7fc4de1cb740 1 -- 172.21.15.107:0/1786137637 wait complete. 2024-04-24T23:47:30.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.622+0000 7fc4de1cb740 1 -- start start 2024-04-24T23:47:30.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 Processor -- start 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdcc40 0x5622d40150b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fe1db0 0x5622d40160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b8010fa0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b80108d0 con 0x5622d3fe1db0 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.623+0000 7fc4de1cb740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b800dbc0 con 0x5622d3fdcc40 2024-04-24T23:47:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.624+0000 7fc4de1cb740 10 client.0 _ll_register_callbacks cb 0x7fc4cc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.627+0000 7fc4d8a5e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdcc40 0x5622d40150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.627+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.627+0000 7fc4d9a60640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fe1db0 0x5622d40160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.627+0000 7fc4d925f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43384/0 (socket says 172.21.15.107:43384) 2024-04-24T23:47:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.627+0000 7fc4d925f640 1 -- 172.21.15.107:0/3056393938 learned_addr learned my addr 172.21.15.107:0/3056393938 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.633+0000 7fc4d925f640 1 -- 172.21.15.107:0/3056393938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdcc40 msgr2=0x5622d40150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.633+0000 7fc4d925f640 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdcc40 0x5622d40150b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.633+0000 7fc4d925f640 1 -- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fe1db0 msgr2=0x5622d40160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.633+0000 7fc4d925f640 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fe1db0 0x5622d40160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.633+0000 7fc4d925f640 1 -- 172.21.15.107:0/3056393938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d40166b0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.639+0000 7fc4d925f640 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0098ca0 tx=0x7fc4d0010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de352b4cf08a5fad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.639+0000 7fc4bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.641+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0098940 con 0x5622d3fd1fe0 2024-04-24T23:47:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.641+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4d00ba070 con 0x5622d3fd1fe0 2024-04-24T23:47:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.641+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc4d00b5210 con 0x5622d3fd1fe0 2024-04-24T23:47:30.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.643+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4d009a5d0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.644+0000 7fc4de1cb740 10 client.16167 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:30.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.644+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622d40179c0 con 0x5622d3fd1fe0 2024-04-24T23:47:30.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d00ba070 con 0x5622d3fd1fe0 2024-04-24T23:47:30.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc4d0098020 con 0x5622d3fd1fe0 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 10 client.16167 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 20 client.16167 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d406a630 0x5622d406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 4 client.16167 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:47:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.645+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5622d3eb46f0 con 0x5622d406a630 2024-04-24T23:47:30.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.649+0000 7fc4d8a5e640 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d406a630 0x5622d406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.666+0000 7fc4d8a5e640 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d406a630 0x5622d406ca50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d454b6c4ba0a221 server_cookie=b922eaf1e2548b0c in_seq=0 out_seq=0 2024-04-24T23:47:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.667+0000 7fc4bcff9640 10 client.16167 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:30.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.670+0000 7fc4bcff9640 1 -- 172.21.15.107:0/3056393938 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5622d3eb46f0 con 0x5622d406a630 2024-04-24T23:47:30.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.670+0000 7fc4bcff9640 10 client.16167 handle_command_reply: tid=0 2024-04-24T23:47:30.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.672+0000 7fc4de1cb740 1 client.16167 shutdown 2024-04-24T23:47:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.675+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 msgr2=0x5622d4014b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.675+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0098ca0 tx=0x7fc4d0010f20 comp rx=0 tx=0).stop 2024-04-24T23:47:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 shutdown_connections 2024-04-24T23:47:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5622d406a630 0x5622d406ca50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5622d3fe1db0 0x5622d40160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5622d3fdcc40 0x5622d40150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 --2- 172.21.15.107:0/3056393938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622d3fd1fe0 0x5622d4014b70 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 >> 172.21.15.107:0/3056393938 conn(0x5622d3fc68b0 msgr2=0x5622d3fc9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.682+0000 7fc4bcff9640 0 client.16167 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.689+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 shutdown_connections 2024-04-24T23:47:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.693+0000 7fc4de1cb740 1 -- 172.21.15.107:0/3056393938 wait complete. 2024-04-24T23:47:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.693+0000 7fc4de1cb740 20 client.16167 trim_cache size 0 max 16384 2024-04-24T23:47:30.699 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:47:30.699 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-04-24T23:47:30.699 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-04-24T23:47:30.699 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:47:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 msgr2=0x7fc4c00a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 0x7fc4c00a51e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5622d3fdd040 tx=0x7fc4c4005b80 comp rx=0 tx=0).stop 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 msgr2=0x7fc4d410e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000b450 tx=0x7fc4d000b910 comp rx=0 tx=0).stop 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 shutdown_connections 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.701+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fc4c00a2d20 0x7fc4c00a51e0 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5622d3fdd040 tx=0x7fc4c4005b80 comp rx=0 tx=0).stop 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.702+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc4d4105330 0x7fc4d410fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.702+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4102980 0x7fc4d410ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.702+0000 7fc4db4c3640 1 --2- 172.21.15.107:0/1033071293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d4100010 0x7fc4d410e4f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.702+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 >> 172.21.15.107:0/1033071293 conn(0x7fc4d40fbc30 msgr2=0x7fc4d40fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.703+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 shutdown_connections 2024-04-24T23:47:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.703+0000 7fc4db4c3640 1 -- 172.21.15.107:0/1033071293 wait complete. 2024-04-24T23:47:30.736 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T23:47:30.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:47:30.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.995+0000 7f9a7a189640 1 -- 172.21.15.107:0/671914079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74101870 msgr2=0x7f9a74101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:30.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:30.995+0000 7f9a7a189640 1 --2- 172.21.15.107:0/671914079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74101870 0x7f9a74101cf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009900 tx=0x7f9a5c02a440 comp rx=0 tx=0).stop 2024-04-24T23:47:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.000+0000 7f9a7a189640 1 -- 172.21.15.107:0/671914079 shutdown_connections 2024-04-24T23:47:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.000+0000 7f9a7a189640 1 --2- 172.21.15.107:0/671914079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a74108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.000+0000 7f9a7a189640 1 --2- 172.21.15.107:0/671914079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74101870 0x7f9a74101cf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.000+0000 7f9a7a189640 1 --2- 172.21.15.107:0/671914079 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a74100680 0x7f9a74100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.000+0000 7f9a7a189640 1 -- 172.21.15.107:0/671914079 >> 172.21.15.107:0/671914079 conn(0x7f9a740fbbf0 msgr2=0x7f9a740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.005+0000 7f9a7a189640 1 -- 172.21.15.107:0/671914079 shutdown_connections 2024-04-24T23:47:31.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.007+0000 7f9a7a189640 1 -- 172.21.15.107:0/671914079 wait complete. 2024-04-24T23:47:31.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.008+0000 7f9a7a189640 1 Processor -- start 2024-04-24T23:47:31.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.008+0000 7f9a7a189640 1 -- start start 2024-04-24T23:47:31.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.008+0000 7f9a7a189640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74100680 0x7f9a740700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.009+0000 7f9a7a189640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a74101870 0x7f9a74070620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.009+0000 7f9a7a189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.009+0000 7f9a7a189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410b290 con 0x7f9a74102230 2024-04-24T23:47:31.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.009+0000 7f9a7a189640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410b110 con 0x7f9a74100680 2024-04-24T23:47:31.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.009+0000 7f9a7a189640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410b410 con 0x7f9a74101870 2024-04-24T23:47:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.010+0000 7f9a73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.010+0000 7f9a73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43398/0 (socket says 172.21.15.107:43398) 2024-04-24T23:47:31.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.010+0000 7f9a73fff640 1 -- 172.21.15.107:0/702881759 learned_addr learned my addr 172.21.15.107:0/702881759 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a737fe640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74100680 0x7f9a740700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a73fff640 1 -- 172.21.15.107:0/702881759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a74101870 msgr2=0x7f9a74070620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a73fff640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a74101870 0x7f9a74070620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a73fff640 1 -- 172.21.15.107:0/702881759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74100680 msgr2=0x7f9a740700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a73fff640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74100680 0x7f9a740700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.013+0000 7f9a73fff640 1 -- 172.21.15.107:0/702881759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a74070d00 con 0x7f9a74102230 2024-04-24T23:47:31.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.014+0000 7f9a73fff640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00b970 tx=0x7f9a6c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.017+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c019070 con 0x7f9a74102230 2024-04-24T23:47:31.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.017+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c00cb40 con 0x7f9a74102230 2024-04-24T23:47:31.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.017+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a741b6fb0 con 0x7f9a74102230 2024-04-24T23:47:31.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.017+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a741b74d0 con 0x7f9a74102230 2024-04-24T23:47:31.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.020+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0076e0 con 0x7f9a74102230 2024-04-24T23:47:31.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.025+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9a6c01c070 con 0x7f9a74102230 2024-04-24T23:47:31.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.026+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a540052c0 con 0x7f9a74102230 2024-04-24T23:47:31.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.032+0000 7f9a70ff9640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 0x7f9a600a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.032+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9a6c0ce8e0 con 0x7f9a74102230 2024-04-24T23:47:31.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.036+0000 7f9a737fe640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 0x7f9a600a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.041+0000 7f9a737fe640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 0x7f9a600a5260 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9a741016d0 tx=0x7f9a64005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.043+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a6c09b410 con 0x7f9a74102230 2024-04-24T23:47:31.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.523+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a540055b0 con 0x7f9a74102230 2024-04-24T23:47:31.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.526+0000 7f9a70ff9640 1 -- 172.21.15.107:0/702881759 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7f9a6c09ad10 con 0x7f9a74102230 2024-04-24T23:47:31.528 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:47:31.529 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:47:31.529 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 msgr2=0x7f9a600a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 0x7f9a600a5260 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9a741016d0 tx=0x7f9a64005c60 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 msgr2=0x7f9a741b69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00b970 tx=0x7f9a6c00be30 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 shutdown_connections 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f9a600a2da0 0x7f9a600a5260 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a74102230 0x7f9a741b69b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f9a74101870 0x7f9a74070620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 --2- 172.21.15.107:0/702881759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a74100680 0x7f9a740700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.532+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 >> 172.21.15.107:0/702881759 conn(0x7f9a740fbbf0 msgr2=0x7f9a74104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.534+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 shutdown_connections 2024-04-24T23:47:31.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.534+0000 7f9a7a189640 1 -- 172.21.15.107:0/702881759 wait complete. 2024-04-24T23:47:31.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:31 smithi195 ceph-mon[33019]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 793 MiB/s wr, 335 op/s 2024-04-24T23:47:31.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:31 smithi107 ceph-mon[29118]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 793 MiB/s wr, 335 op/s 2024-04-24T23:47:31.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-24T23:47:31.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:31 smithi170 ceph-mon[32714]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 793 MiB/s wr, 335 op/s 2024-04-24T23:47:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.809+0000 7f8a027fc640 1 -- 172.21.15.107:0/1930643000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8030920 con 0x7f8a04072b80 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 -- 172.21.15.107:0/1930643000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04072b80 msgr2=0x7f8a04072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1930643000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04072b80 0x7f8a04072fa0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009860 tx=0x7f89f802a360 comp rx=0 tx=0).stop 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 -- 172.21.15.107:0/1930643000 shutdown_connections 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1930643000 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a041097c0 0x7f8a0410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1930643000 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a04108e20 0x7f8a04109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1930643000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04072b80 0x7f8a04072fa0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 -- 172.21.15.107:0/1930643000 >> 172.21.15.107:0/1930643000 conn(0x7f8a0406e650 msgr2=0x7f8a04070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 -- 172.21.15.107:0/1930643000 shutdown_connections 2024-04-24T23:47:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.810+0000 7f8a088d9640 1 -- 172.21.15.107:0/1930643000 wait complete. 2024-04-24T23:47:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.811+0000 7f8a088d9640 1 Processor -- start 2024-04-24T23:47:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.812+0000 7f8a088d9640 1 -- start start 2024-04-24T23:47:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.812+0000 7f8a088d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a088d9640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a041097c0 0x7f8a041a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a088d9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a041abee0 0x7f8a041ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a088d9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0411c120 con 0x7f8a04108e20 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a088d9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0411bfa0 con 0x7f8a041abee0 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a088d9640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0411c2a0 con 0x7f8a041097c0 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43410/0 (socket says 172.21.15.107:43410) 2024-04-24T23:47:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.813+0000 7f8a037fe640 1 -- 172.21.15.107:0/1652162855 learned_addr learned my addr 172.21.15.107:0/1652162855 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 -- 172.21.15.107:0/1652162855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a041097c0 msgr2=0x7f8a041a6e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a041097c0 0x7f8a041a6e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 -- 172.21.15.107:0/1652162855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a041abee0 msgr2=0x7f8a041ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a041abee0 0x7f8a041ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 -- 172.21.15.107:0/1652162855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a041ae940 con 0x7f8a04108e20 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f89f8030580 tx=0x7f89f8033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a00ff9640 1 -- 172.21.15.107:0/1652162855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8037040 con 0x7f8a04108e20 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.814+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a041aebd0 con 0x7f8a04108e20 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.815+0000 7f8a00ff9640 1 -- 172.21.15.107:0/1652162855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8004040 con 0x7f8a04108e20 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.815+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a041af090 con 0x7f8a04108e20 2024-04-24T23:47:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.815+0000 7f8a00ff9640 1 -- 172.21.15.107:0/1652162855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8003990 con 0x7f8a04108e20 2024-04-24T23:47:31.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.819+0000 7f8a00ff9640 1 -- 172.21.15.107:0/1652162855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f89f8044030 con 0x7f8a04108e20 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 Processor -- start 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 -- start start 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a00ff9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 0x7f89ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba8d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a00ff9640 1 -- 172.21.15.107:0/1652162855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f89f80eeff0 con 0x7f8a04108e20 2024-04-24T23:47:31.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ecb9ec670 con 0x560ecba8df20 2024-04-24T23:47:31.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ecb994970 con 0x560ecba883c0 2024-04-24T23:47:31.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.821+0000 7f8a0c846740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ecb9e8630 con 0x560ecba88db0 2024-04-24T23:47:31.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.824+0000 7f8a037fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.824+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.824+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43420/0 (socket says 172.21.15.107:43420) 2024-04-24T23:47:31.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.824+0000 7f8a03fff640 1 -- 172.21.15.107:0/1550813374 learned_addr learned my addr 172.21.15.107:0/1550813374 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.824+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 0x7f89ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.825+0000 7f8a03fff640 1 -- 172.21.15.107:0/1550813374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 msgr2=0x560ecba8d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.825+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba8d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.825+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba8d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.825+0000 7f8a03fff640 1 -- 172.21.15.107:0/1550813374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 msgr2=0x560ecba887e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.825+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba887e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.826+0000 7f8a03fff640 1 -- 172.21.15.107:0/1550813374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ecb9b8400 con 0x560ecba8df20 2024-04-24T23:47:31.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.826+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba8d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:31.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.826+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 0x7f89ec0a50d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009270 tx=0x7f89fc00c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.827+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f89fc012d20 tx=0x7f89fc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a66b3f09000e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.829+0000 7f89e9ffb640 1 -- 172.21.15.107:0/1550813374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc020070 con 0x560ecba8df20 2024-04-24T23:47:31.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.829+0000 7f89e9ffb640 1 -- 172.21.15.107:0/1550813374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89fc009b10 con 0x560ecba8df20 2024-04-24T23:47:31.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.829+0000 7f89e9ffb640 1 -- 172.21.15.107:0/1550813374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc01cdb0 con 0x560ecba8df20 2024-04-24T23:47:31.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 -- 172.21.15.107:0/1550813374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 msgr2=0x560ecba903b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f89fc012d20 tx=0x7f89fc00f730 comp rx=0 tx=0).stop 2024-04-24T23:47:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 -- 172.21.15.107:0/1550813374 shutdown_connections 2024-04-24T23:47:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba903b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba8d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1550813374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba887e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.830+0000 7f8a0c846740 1 -- 172.21.15.107:0/1550813374 >> 172.21.15.107:0/1550813374 conn(0x560ecba83a30 msgr2=0x560ecba83e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.831+0000 7f8a0c846740 1 -- 172.21.15.107:0/1550813374 shutdown_connections 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.831+0000 7f8a0c846740 1 -- 172.21.15.107:0/1550813374 wait complete. 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.831+0000 7f8a0c846740 1 -- start start 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.832+0000 7f8a0c846740 1 Processor -- start 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.832+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba92510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.832+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba92a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.832+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a0c846740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560ecba90e70 con 0x560ecba8df20 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a0c846740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x560ecb9a8050 con 0x560ecba883c0 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a0c846740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560ecba90ff0 con 0x560ecba88db0 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a0c846740 10 client.0 _ll_register_callbacks cb 0x560ecb6ffbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43430/0 (socket says 172.21.15.107:43430) 2024-04-24T23:47:31.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.833+0000 7f8a03fff640 1 -- 172.21.15.107:0/1796153854 learned_addr learned my addr 172.21.15.107:0/1796153854 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.834+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba92a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba92510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a03fff640 1 -- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 msgr2=0x560ecba92a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba92a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a03fff640 1 -- 172.21.15.107:0/1796153854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 msgr2=0x560ecba92510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba92510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.837+0000 7f8a03fff640 1 -- 172.21.15.107:0/1796153854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ecb979dc0 con 0x560ecba8df20 2024-04-24T23:47:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.838+0000 7f8a03fff640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00bd50 tx=0x7f89fc00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef713aa027e6dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.840+0000 7f89e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:31.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.840+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc020050 con 0x560ecba8df20 2024-04-24T23:47:31.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.841+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f89fc01ccc0 con 0x560ecba8df20 2024-04-24T23:47:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.841+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560ecb9f9bb0 con 0x560ecba8df20 2024-04-24T23:47:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.842+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89fc005bd0 con 0x560ecba8df20 2024-04-24T23:47:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.842+0000 7f8a0c846740 10 client.16182 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:31.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.842+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560ecb7003f0 con 0x560ecba8df20 2024-04-24T23:47:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560ecb9f9bb0 con 0x560ecba8df20 2024-04-24T23:47:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f89fc02d430 con 0x560ecba8df20 2024-04-24T23:47:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 10 client.16182 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:31.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 10 client.16182 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:47:31.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 20 client.16182 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:31.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecba9e230 0x560ecbaa0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 4 client.16182 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:31.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.843+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ecb9f3f00 con 0x560ecba9e230 2024-04-24T23:47:31.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.845+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecba9e230 0x560ecbaa0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.853+0000 7f8a037fe640 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecba9e230 0x560ecbaa0690 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b4d43b99dfdbcf31 server_cookie=7eeb259f043b0764 in_seq=0 out_seq=0 2024-04-24T23:47:31.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.853+0000 7f89e2ffd640 10 client.16182 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:47:31.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.855+0000 7f89e2ffd640 1 -- 172.21.15.107:0/1796153854 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ecb9f3f00 con 0x560ecba9e230 2024-04-24T23:47:31.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.855+0000 7f89e2ffd640 10 client.16182 handle_command_reply: tid=0 2024-04-24T23:47:31.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.855+0000 7f8a0c846740 1 client.16182 shutdown 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.856+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 msgr2=0x560ecba99a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.856+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00bd50 tx=0x7f89fc00bd80 comp rx=0 tx=0).stop 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 shutdown_connections 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecba9e230 0x560ecbaa0690 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba99a90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba92a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 --2- 172.21.15.107:0/1796153854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba92510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 >> 172.21.15.107:0/1796153854 conn(0x560ecba83a30 msgr2=0x560ecba82b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.857+0000 7f89e2ffd640 0 client.16182 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:47:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.859+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 shutdown_connections 2024-04-24T23:47:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.859+0000 7f8a0c846740 1 -- 172.21.15.107:0/1796153854 wait complete. 2024-04-24T23:47:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.861+0000 7f8a0c846740 20 client.16182 trim_cache size 0 max 16384 2024-04-24T23:47:31.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.903+0000 7f8a0c846740 1 Processor -- start 2024-04-24T23:47:31.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 -- start start 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba89850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba8df20 0x560ecba89d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89f000fd90 con 0x560ecba88db0 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89f00118d0 con 0x560ecba883c0 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.904+0000 7f8a0c846740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f89f0010460 con 0x560ecba8df20 2024-04-24T23:47:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.905+0000 7f8a02ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43434/0 (socket says 172.21.15.107:43434) 2024-04-24T23:47:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 -- 172.21.15.107:0/2115370527 learned_addr learned my addr 172.21.15.107:0/2115370527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 -- 172.21.15.107:0/2115370527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba8df20 msgr2=0x560ecba89d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:47:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba8df20 0x560ecba89d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 -- 172.21.15.107:0/2115370527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 msgr2=0x560ecba89850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba89850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.906+0000 7f8a02ffd640 1 -- 172.21.15.107:0/2115370527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ecb7ca7d0 con 0x560ecba88db0 2024-04-24T23:47:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.908+0000 7f8a02ffd640 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8a041a7e50 tx=0x7f89f40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9afc0c09c98b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.910+0000 7f89e97fa640 1 -- 172.21.15.107:0/2115370527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4004030 con 0x560ecba88db0 2024-04-24T23:47:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.910+0000 7f89e97fa640 1 -- 172.21.15.107:0/2115370527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f40041d0 con 0x560ecba88db0 2024-04-24T23:47:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.911+0000 7f8a0c846740 1 -- 172.21.15.107:0/2115370527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 msgr2=0x560ecba99a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.911+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8a041a7e50 tx=0x7f89f40117b0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.912+0000 7f8a0c846740 1 -- 172.21.15.107:0/2115370527 shutdown_connections 2024-04-24T23:47:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.912+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba8df20 0x560ecba89d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.912+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba88db0 0x560ecba99a50 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.912+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2115370527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba89850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.912+0000 7f8a0c846740 1 -- 172.21.15.107:0/2115370527 >> 172.21.15.107:0/2115370527 conn(0x560ecba83a30 msgr2=0x560ecba82b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.913+0000 7f8a0c846740 1 -- 172.21.15.107:0/2115370527 shutdown_connections 2024-04-24T23:47:31.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.914+0000 7f8a0c846740 1 -- 172.21.15.107:0/2115370527 wait complete. 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.914+0000 7f8a0c846740 1 -- start start 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 Processor -- start 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba96190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89f0011200 con 0x560ecba8df20 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89f0010b30 con 0x560ecba883c0 2024-04-24T23:47:31.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f89f000dd70 con 0x560ecba88db0 2024-04-24T23:47:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.915+0000 7f8a0c846740 10 client.0 _ll_register_callbacks cb 0x7f89fc012b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43440/0 (socket says 172.21.15.107:43440) 2024-04-24T23:47:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 -- 172.21.15.107:0/2848583329 learned_addr learned my addr 172.21.15.107:0/2848583329 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 -- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 msgr2=0x560ecba96190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba96190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 -- 172.21.15.107:0/2848583329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 msgr2=0x560ecba996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.916+0000 7f8a03fff640 1 -- 172.21.15.107:0/2848583329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ecba96c40 con 0x560ecba8df20 2024-04-24T23:47:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.917+0000 7f8a03fff640 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89fc005d70 tx=0x7f89fc01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2cf6dde754bdca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.918+0000 7f89e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.918+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc020040 con 0x560ecba8df20 2024-04-24T23:47:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.918+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f89fc012680 con 0x560ecba8df20 2024-04-24T23:47:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.918+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f40041d0 con 0x560ecba8df20 2024-04-24T23:47:31.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.923+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89fc045050 con 0x560ecba8df20 2024-04-24T23:47:31.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.923+0000 7f8a0c846740 10 client.16188 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:31.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.923+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560ecbb14900 con 0x560ecba8df20 2024-04-24T23:47:31.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.925+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f40041d0 con 0x560ecba8df20 2024-04-24T23:47:31.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.925+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f89fc004070 con 0x560ecba8df20 2024-04-24T23:47:31.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 10 client.16188 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:31.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 10 client.16188 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:47:31.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 20 client.16188 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:31.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecbb15310 0x560ecbb17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:31.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 4 client.16188 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:47:31.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.926+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ecb971820 con 0x560ecbb15310 2024-04-24T23:47:31.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.927+0000 7f8a037fe640 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecbb15310 0x560ecbb17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.934+0000 7f8a037fe640 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecbb15310 0x560ecbb17730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8689593e2c60db77 server_cookie=a8f463ace5a91181 in_seq=0 out_seq=0 2024-04-24T23:47:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.934+0000 7f89e2ffd640 10 client.16188 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:47:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.934+0000 7f89e2ffd640 1 -- 172.21.15.107:0/2848583329 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ecb971820 con 0x560ecbb15310 2024-04-24T23:47:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.934+0000 7f89e2ffd640 10 client.16188 handle_command_reply: tid=0 2024-04-24T23:47:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.934+0000 7f8a0c846740 1 client.16188 shutdown 2024-04-24T23:47:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 msgr2=0x560ecba966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89fc005d70 tx=0x7f89fc01c6a0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 shutdown_connections 2024-04-24T23:47:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x560ecbb15310 0x560ecbb17730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560ecba8df20 0x560ecba966d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x560ecba88db0 0x560ecba96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 --2- 172.21.15.107:0/2848583329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560ecba883c0 0x560ecba996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 >> 172.21.15.107:0/2848583329 conn(0x560ecba83a30 msgr2=0x560ecba913f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.935+0000 7f89e2ffd640 0 client.16188 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.936+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 shutdown_connections 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.936+0000 7f8a0c846740 1 -- 172.21.15.107:0/2848583329 wait complete. 2024-04-24T23:47:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.936+0000 7f8a0c846740 20 client.16188 trim_cache size 0 max 16384 2024-04-24T23:47:31.937 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:47:31.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.938+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 msgr2=0x7f89ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 0x7f89ec0a50d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009270 tx=0x7f89fc00c000 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 msgr2=0x7f8a041a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f89f8030580 tx=0x7f89f8033d90 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 shutdown_connections 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f89ec0a2c10 0x7f89ec0a50d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a041abee0 0x7f8a041ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8a041097c0 0x7f8a041a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 --2- 172.21.15.107:0/1652162855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a04108e20 0x7f8a041a6940 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 >> 172.21.15.107:0/1652162855 conn(0x7f8a0406e650 msgr2=0x7f8a040702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.939+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 shutdown_connections 2024-04-24T23:47:31.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:31.940+0000 7f8a088d9640 1 -- 172.21.15.107:0/1652162855 wait complete. 2024-04-24T23:47:31.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.155+0000 7f82699b0640 1 -- 172.21.15.107:0/4132870580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c097960 msgr2=0x7f825c09e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.155+0000 7f82699b0640 1 --2- 172.21.15.107:0/4132870580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c097960 0x7f825c09e1f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8264066ea0 tx=0x7f8264096040 comp rx=0 tx=0).stop 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 -- 172.21.15.107:0/4132870580 shutdown_connections 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 --2- 172.21.15.107:0/4132870580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c097960 0x7f825c09e1f0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 --2- 172.21.15.107:0/4132870580 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f825c096fa0 0x7f825c097420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 --2- 172.21.15.107:0/4132870580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 0x7f825c0961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 -- 172.21.15.107:0/4132870580 >> 172.21.15.107:0/4132870580 conn(0x7f825c091340 msgr2=0x7f825c093760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 -- 172.21.15.107:0/4132870580 shutdown_connections 2024-04-24T23:47:32.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.156+0000 7f82699b0640 1 -- 172.21.15.107:0/4132870580 wait complete. 2024-04-24T23:47:32.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.157+0000 7f82699b0640 1 Processor -- start 2024-04-24T23:47:32.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.158+0000 7f82699b0640 1 -- start start 2024-04-24T23:47:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.158+0000 7f82699b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 0x7f825c12bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.158+0000 7f82699b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82699b0640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f825c097960 0x7f825c133310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43464/0 (socket says 172.21.15.107:43464) 2024-04-24T23:47:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82627fc640 1 -- 172.21.15.107:0/1911964387 learned_addr learned my addr 172.21.15.107:0/1911964387 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0a1960 con 0x7f825c096fa0 2024-04-24T23:47:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0a17e0 con 0x7f825c095db0 2024-04-24T23:47:32.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.159+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0a1ae0 con 0x7f825c097960 2024-04-24T23:47:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f8262ffd640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 0x7f825c12bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f82627fc640 1 -- 172.21.15.107:0/1911964387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f825c097960 msgr2=0x7f825c133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f82627fc640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f825c097960 0x7f825c133310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f82627fc640 1 -- 172.21.15.107:0/1911964387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 msgr2=0x7f825c12bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f82627fc640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 0x7f825c12bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.160+0000 7f82627fc640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c133980 con 0x7f825c096fa0 2024-04-24T23:47:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.161+0000 7f82627fc640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f825800bea0 tx=0x7f825800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.162+0000 7f82689ae640 1 -- 172.21.15.107:0/1911964387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258018070 con 0x7f825c096fa0 2024-04-24T23:47:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.162+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c007710 con 0x7f825c096fa0 2024-04-24T23:47:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.162+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c007cb0 con 0x7f825c096fa0 2024-04-24T23:47:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.163+0000 7f82689ae640 1 -- 172.21.15.107:0/1911964387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258012c40 con 0x7f825c096fa0 2024-04-24T23:47:32.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.163+0000 7f82689ae640 1 -- 172.21.15.107:0/1911964387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825801b9a0 con 0x7f825c096fa0 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.167+0000 7f826c79e740 1 Processor -- start 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.167+0000 7f826c79e740 1 -- start start 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f82689ae640 1 -- 172.21.15.107:0/1911964387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f825801bb40 con 0x7f825c096fa0 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56046de91670 con 0x56046df43f40 2024-04-24T23:47:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56046de39970 con 0x56046df3e3e0 2024-04-24T23:47:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.168+0000 7f826c79e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56046de8d630 con 0x56046df3edd0 2024-04-24T23:47:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.169+0000 7f82637fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.169+0000 7f82637fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:43908/0 (socket says 172.21.15.107:43908) 2024-04-24T23:47:32.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.169+0000 7f82637fe640 1 -- 172.21.15.107:0/1224839094 learned_addr learned my addr 172.21.15.107:0/1224839094 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.170+0000 7f82689ae640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 0x7f824c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.170+0000 7f82627fc640 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.170+0000 7f82689ae640 1 -- 172.21.15.107:0/1911964387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f82580d3910 con 0x7f825c096fa0 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f8262ffd640 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 -- 172.21.15.107:0/1224839094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 msgr2=0x56046df438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 -- 172.21.15.107:0/1224839094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 msgr2=0x56046df463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df463d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 -- 172.21.15.107:0/1224839094 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56046de5d400 con 0x56046df3e3e0 2024-04-24T23:47:32.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.171+0000 7f82637fe640 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8264000b60 tx=0x7f826404fe20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa964830366c5ed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.172+0000 7f82637fe640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 0x7f824c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.172+0000 7f8249ffb640 1 -- 172.21.15.107:0/1224839094 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640735f0 con 0x56046df3e3e0 2024-04-24T23:47:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.172+0000 7f8249ffb640 1 -- 172.21.15.107:0/1224839094 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8264073790 con 0x56046df3e3e0 2024-04-24T23:47:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.172+0000 7f8249ffb640 1 -- 172.21.15.107:0/1224839094 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640a7410 con 0x56046df3e3e0 2024-04-24T23:47:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.173+0000 7f82637fe640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 0x7f824c0a5150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8264069be0 tx=0x7f82640a7690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.174+0000 7f826c79e740 1 -- 172.21.15.107:0/1224839094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 msgr2=0x56046df3e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.174+0000 7f826c79e740 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8264000b60 tx=0x7f826404fe20 comp rx=0 tx=0).stop 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 -- 172.21.15.107:0/1224839094 shutdown_connections 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 --2- 172.21.15.107:0/1224839094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df3e800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 -- 172.21.15.107:0/1224839094 >> 172.21.15.107:0/1224839094 conn(0x56046df28a40 msgr2=0x56046df28e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 -- 172.21.15.107:0/1224839094 shutdown_connections 2024-04-24T23:47:32.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.175+0000 7f826c79e740 1 -- 172.21.15.107:0/1224839094 wait complete. 2024-04-24T23:47:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.176+0000 7f826c79e740 1 -- start start 2024-04-24T23:47:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.176+0000 7f826c79e740 1 Processor -- start 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3e3e0 0x56046df48530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56046df46e90 con 0x56046df3e3e0 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56046de4d050 con 0x56046df3edd0 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f8262ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3e3e0 0x56046df48530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x56046df47010 con 0x56046df43f40 2024-04-24T23:47:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f826c79e740 10 client.0 _ll_register_callbacks cb 0x56046dba4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f82627fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f82627fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:43910/0 (socket says 172.21.15.107:43910) 2024-04-24T23:47:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.177+0000 7f82627fc640 1 -- 172.21.15.107:0/404158215 learned_addr learned my addr 172.21.15.107:0/404158215 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82637fe640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 -- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 msgr2=0x56046df4fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 -- 172.21.15.107:0/404158215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3e3e0 msgr2=0x56046df48530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3e3e0 0x56046df48530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 -- 172.21.15.107:0/404158215 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56046de1edc0 con 0x56046df3edd0 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f82627fc640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82580a8480 tx=0x7f825809f760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a999ee43cb6e252b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.178+0000 7f8242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-04-24T23:47:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825809fd40 con 0x56046df3edd0 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mon.2 v2:172.21.15.170:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8258009c30 con 0x56046df3edd0 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8264073790 con 0x56046df3edd0 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82580a40f0 con 0x56046df3edd0 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f826c79e740 10 client.25541 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.179+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56046dba53f0 con 0x56046df3edd0 2024-04-24T23:47:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mon.2 v2:172.21.15.170:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82580a89b0 con 0x56046df3edd0 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mon.2 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82580b6020 con 0x56046df3edd0 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 10 client.25541 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 10 client.25541 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 20 client.25541 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046df54380 0x56046df56800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 4 client.25541 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.180+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8264073790 con 0x56046df54380 2024-04-24T23:47:32.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.181+0000 7f8262ffd640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046df54380 0x56046df56800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.188+0000 7f8262ffd640 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046df54380 0x56046df56800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4feca4cab2bc9a9d server_cookie=1cd971d71c1c8e0a in_seq=0 out_seq=0 2024-04-24T23:47:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.188+0000 7f8242ffd640 10 client.25541 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:47:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.190+0000 7f8242ffd640 1 -- 172.21.15.107:0/404158215 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8264073790 con 0x56046df54380 2024-04-24T23:47:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.190+0000 7f8242ffd640 10 client.25541 handle_command_reply: tid=0 2024-04-24T23:47:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 client.25541 shutdown 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 msgr2=0x56046df48a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82580a8480 tx=0x7f825809f760 comp rx=0 tx=0).stop 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 shutdown_connections 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046df54380 0x56046df56800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3edd0 0x56046df48a70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 --2- 172.21.15.107:0/404158215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3e3e0 0x56046df48530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.191+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 >> 172.21.15.107:0/404158215 conn(0x56046df28a40 msgr2=0x56046df27bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.192+0000 7f8242ffd640 0 client.25541 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.192+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 shutdown_connections 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.192+0000 7f826c79e740 1 -- 172.21.15.107:0/404158215 wait complete. 2024-04-24T23:47:32.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.192+0000 7f826c79e740 20 client.25541 trim_cache size 0 max 16384 2024-04-24T23:47:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.217+0000 7f826c79e740 1 Processor -- start 2024-04-24T23:47:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.217+0000 7f826c79e740 1 -- start start 2024-04-24T23:47:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.217+0000 7f826c79e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df23c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.217+0000 7f826c79e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df4fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f826c79e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f826c79e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f825000fbd0 con 0x56046df43f40 2024-04-24T23:47:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f826c79e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8250011710 con 0x56046df3e3e0 2024-04-24T23:47:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f826c79e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f82500102a0 con 0x56046df3edd0 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f82637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f82637fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43490/0 (socket says 172.21.15.107:43490) 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.218+0000 7f82637fe640 1 -- 172.21.15.107:0/481556483 learned_addr learned my addr 172.21.15.107:0/481556483 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82627fc640 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df4fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82637fe640 1 -- 172.21.15.107:0/481556483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 msgr2=0x56046df4fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82637fe640 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df4fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82637fe640 1 -- 172.21.15.107:0/481556483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 msgr2=0x56046df23c00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:47:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82637fe640 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df23c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.219+0000 7f82637fe640 1 -- 172.21.15.107:0/481556483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56046dc6f7d0 con 0x56046df43f40 2024-04-24T23:47:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.220+0000 7f82637fe640 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f826404feb0 tx=0x7f82640ac000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85e45f3690e8cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.221+0000 7f82497fa640 1 -- 172.21.15.107:0/481556483 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640a7410 con 0x56046df43f40 2024-04-24T23:47:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.222+0000 7f82497fa640 1 -- 172.21.15.107:0/481556483 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826409d6b0 con 0x56046df43f40 2024-04-24T23:47:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.222+0000 7f82497fa640 1 -- 172.21.15.107:0/481556483 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826409e7d0 con 0x56046df43f40 2024-04-24T23:47:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.223+0000 7f826c79e740 1 -- 172.21.15.107:0/481556483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 msgr2=0x56046df3f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.223+0000 7f826c79e740 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f826404feb0 tx=0x7f82640ac000 comp rx=0 tx=0).stop 2024-04-24T23:47:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.225+0000 7f826c79e740 1 -- 172.21.15.107:0/481556483 shutdown_connections 2024-04-24T23:47:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.225+0000 7f826c79e740 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df43f40 0x56046df3f870 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.225+0000 7f826c79e740 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df3edd0 0x56046df4fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.225+0000 7f826c79e740 1 --2- 172.21.15.107:0/481556483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df23c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.225+0000 7f826c79e740 1 -- 172.21.15.107:0/481556483 >> 172.21.15.107:0/481556483 conn(0x56046df28a40 msgr2=0x56046df2bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.229+0000 7f826c79e740 1 -- 172.21.15.107:0/481556483 shutdown_connections 2024-04-24T23:47:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.230+0000 7f826c79e740 1 -- 172.21.15.107:0/481556483 wait complete. 2024-04-24T23:47:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.230+0000 7f826c79e740 1 -- start start 2024-04-24T23:47:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.230+0000 7f826c79e740 1 Processor -- start 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df4d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8250011040 con 0x56046df3edd0 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8250010970 con 0x56046df3e3e0 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f825000dbb0 con 0x56046df43f40 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f826c79e740 10 client.0 _ll_register_callbacks cb 0x7f825801b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f8262ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df4d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f82627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f82627fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43502/0 (socket says 172.21.15.107:43502) 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.231+0000 7f82627fc640 1 -- 172.21.15.107:0/1485788414 learned_addr learned my addr 172.21.15.107:0/1485788414 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.232+0000 7f82627fc640 1 -- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 msgr2=0x56046df4ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.232+0000 7f82627fc640 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.232+0000 7f82627fc640 1 -- 172.21.15.107:0/1485788414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 msgr2=0x56046df4d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.232+0000 7f82627fc640 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df4d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.232+0000 7f82627fc640 1 -- 172.21.15.107:0/1485788414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56046df4f020 con 0x56046df3edd0 2024-04-24T23:47:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.233+0000 7f82627fc640 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f825809fa40 tx=0x7f825809c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1facc1128ee6341 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.233+0000 7f8242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.240+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825809f880 con 0x56046df3edd0 2024-04-24T23:47:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.240+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8258012a10 con 0x56046df3edd0 2024-04-24T23:47:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.240+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f826409d6b0 con 0x56046df3edd0 2024-04-24T23:47:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.241+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82580a89b0 con 0x56046df3edd0 2024-04-24T23:47:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.241+0000 7f826c79e740 10 client.16203 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.241+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56046dfc3440 con 0x56046df3edd0 2024-04-24T23:47:32.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.248+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825809fd40 con 0x56046df3edd0 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82580b6070 con 0x56046df3edd0 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 10 client.16203 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 10 client.16203 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 20 client.16203 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046dfc3df0 0x56046dfc6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 4 client.16203 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:47:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.249+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f826409d6b0 con 0x56046dfc3df0 2024-04-24T23:47:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.251+0000 7f8262ffd640 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046dfc3df0 0x56046dfc6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.253+0000 7f8242ffd640 10 client.16203 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:47:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.253+0000 7f8262ffd640 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046dfc3df0 0x56046dfc6210 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f8e4c32c0203c028 server_cookie=84031b4bbfdfd947 in_seq=0 out_seq=0 2024-04-24T23:47:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.256+0000 7f8242ffd640 1 -- 172.21.15.107:0/1485788414 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f826409d6b0 con 0x56046dfc3df0 2024-04-24T23:47:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.256+0000 7f8242ffd640 10 client.16203 handle_command_reply: tid=0 2024-04-24T23:47:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.256+0000 7f826c79e740 1 client.16203 shutdown 2024-04-24T23:47:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.256+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 msgr2=0x56046df4da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.256+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f825809fa40 tx=0x7f825809c5e0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 shutdown_connections 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x56046dfc3df0 0x56046dfc6210 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x56046df43f40 0x56046df4ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f8242ffd640 0 client.16203 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56046df3edd0 0x56046df4da20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 --2- 172.21.15.107:0/1485788414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56046df3e3e0 0x56046df4d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 >> 172.21.15.107:0/1485788414 conn(0x56046df28a40 msgr2=0x56046df27330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 shutdown_connections 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.257+0000 7f826c79e740 1 -- 172.21.15.107:0/1485788414 wait complete. 2024-04-24T23:47:32.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f826c79e740 20 client.16203 trim_cache size 0 max 16384 2024-04-24T23:47:32.259 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:47:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 msgr2=0x7f824c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 0x7f824c0a5150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8264069be0 tx=0x7f82640a7690 comp rx=0 tx=0).stop 2024-04-24T23:47:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 msgr2=0x7f825c12c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f825800bea0 tx=0x7f825800bf90 comp rx=0 tx=0).stop 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 shutdown_connections 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f824c0a2c90 0x7f824c0a5150 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f825c097960 0x7f825c133310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f825c096fa0 0x7f825c12c2f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 --2- 172.21.15.107:0/1911964387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f825c095db0 0x7f825c12bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.258+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 >> 172.21.15.107:0/1911964387 conn(0x7f825c091340 msgr2=0x7f825c09a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.259+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 shutdown_connections 2024-04-24T23:47:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.259+0000 7f82699b0640 1 -- 172.21.15.107:0/1911964387 wait complete. 2024-04-24T23:47:32.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 rewriters = 985988.75 kB/sec 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 rewriters = 985967.18 kB/sec 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 985988.75 kB/sec 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 985988.75 kB/sec 2024-04-24T23:47:32.409 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 985988.75 kB/sec 2024-04-24T23:47:32.410 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 10485760.00 kB 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- 172.21.15.107:0/3829797591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8101870 msgr2=0x7f24d8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 --2- 172.21.15.107:0/3829797591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8101870 0x7f24d8101cf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009900 tx=0x7f24c802a410 comp rx=0 tx=0).stop 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- 172.21.15.107:0/3829797591 shutdown_connections 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 --2- 172.21.15.107:0/3829797591 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8102230 0x7f24d8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 --2- 172.21.15.107:0/3829797591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8101870 0x7f24d8101cf0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 --2- 172.21.15.107:0/3829797591 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 0x7f24d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- 172.21.15.107:0/3829797591 >> 172.21.15.107:0/3829797591 conn(0x7f24d80fbbf0 msgr2=0x7f24d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- 172.21.15.107:0/3829797591 shutdown_connections 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- 172.21.15.107:0/3829797591 wait complete. 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 Processor -- start 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.433+0000 7f24dff48640 1 -- start start 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 0x7f24d80694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8101870 0x7f24d819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810c330 con 0x7f24d8102230 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810c1b0 con 0x7f24d8101870 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810c4b0 con 0x7f24d8100680 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43508/0 (socket says 172.21.15.107:43508) 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 -- 172.21.15.107:0/1383998469 learned_addr learned my addr 172.21.15.107:0/1383998469 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8101870 0x7f24d819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24ddce4640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 0x7f24d80694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 -- 172.21.15.107:0/1383998469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 msgr2=0x7f24d80694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 0x7f24d80694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 -- 172.21.15.107:0/1383998469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8101870 msgr2=0x7f24d819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8101870 0x7f24d819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 -- 172.21.15.107:0/1383998469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d819f730 con 0x7f24d8102230 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24de4e5640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f24d000cad0 tx=0x7f24d000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24ceffd640 1 -- 172.21.15.107:0/1383998469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d000ae20 con 0x7f24d8102230 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24ceffd640 1 -- 172.21.15.107:0/1383998469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d0005020 con 0x7f24d8102230 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d819fa20 con 0x7f24d8102230 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24ceffd640 1 -- 172.21.15.107:0/1383998469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d000f6f0 con 0x7f24d8102230 2024-04-24T23:47:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.434+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d819fef0 con 0x7f24d8102230 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24ceffd640 1 -- 172.21.15.107:0/1383998469 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f24d00051d0 con 0x7f24d8102230 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 Processor -- start 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 -- start start 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd57fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd582630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd4da670 con 0x55fcdd57a640 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd482970 con 0x55fcdd57b030 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd4d6630 con 0x55fcdd5801a0 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24ceffd640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 0x7f24c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43522/0 (socket says 172.21.15.107:43522) 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24dd4e3640 1 -- 172.21.15.107:0/2216208587 learned_addr learned my addr 172.21.15.107:0/2216208587 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.437+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd57fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd582630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24de4e5640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 0x7f24c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 -- 172.21.15.107:0/2216208587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 msgr2=0x55fcdd582630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd582630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 -- 172.21.15.107:0/2216208587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 msgr2=0x55fcdd57aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 -- 172.21.15.107:0/2216208587 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdd4a6400 con 0x55fcdd57b030 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ceffd640 1 -- 172.21.15.107:0/1383998469 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f24d00cf4d0 con 0x7f24d8102230 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd57fb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24d400bbb0 tx=0x7f24d400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6431683ebea1cc95 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24c3fff640 1 -- 172.21.15.107:0/2216208587 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4015070 con 0x55fcdd57b030 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24de4e5640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 0x7f24c40a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24d009b660 tx=0x7f24d006b810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24c3fff640 1 -- 172.21.15.107:0/2216208587 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4005020 con 0x55fcdd57b030 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.438+0000 7f24c3fff640 1 -- 172.21.15.107:0/2216208587 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4004a00 con 0x55fcdd57b030 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2216208587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 msgr2=0x55fcdd57fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd57fb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24d400bbb0 tx=0x7f24d400f730 comp rx=0 tx=0).stop 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2216208587 shutdown_connections 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd582630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd57fb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2216208587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd57aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2216208587 >> 172.21.15.107:0/2216208587 conn(0x55fcdd5718a0 msgr2=0x55fcdd571ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2216208587 shutdown_connections 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2216208587 wait complete. 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 -- start start 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.439+0000 7f24e2c4c740 1 Processor -- start 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57a640 0x55fcdd584730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57b030 0x55fcdd584c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd583150 con 0x55fcdd5801a0 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd496050 con 0x55fcdd57a640 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55fcdd5832d0 con 0x55fcdd57b030 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 10 client.0 _ll_register_callbacks cb 0x55fcdd1edbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43526/0 (socket says 172.21.15.107:43526) 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 -- 172.21.15.107:0/1976380060 learned_addr learned my addr 172.21.15.107:0/1976380060 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57a640 0x55fcdd584730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24ddce4640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57b030 0x55fcdd584c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 -- 172.21.15.107:0/1976380060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57b030 msgr2=0x55fcdd584c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57b030 0x55fcdd584c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 -- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57a640 msgr2=0x55fcdd584730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57a640 0x55fcdd584730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 -- 172.21.15.107:0/1976380060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdd467dc0 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f24c8031440 tx=0x7f24c80315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a929b3263cbcf5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035040 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f24c8032dd0 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24d4005020 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24c8005020 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 10 client.16215 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.440+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcdd1ee3f0 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d4005020 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24c80035f0 con 0x55fcdd5801a0 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 10 client.16215 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 10 client.16215 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 20 client.16215 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5904e0 0x55fcdd592950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 4 client.16215 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcdd4e7bb0 con 0x55fcdd5904e0 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24ddce4640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5904e0 0x55fcdd592950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24ddce4640 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5904e0 0x55fcdd592950 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bcd6f92217a37a5e server_cookie=ca76876ead94119b in_seq=0 out_seq=0 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.441+0000 7f24c17fa640 10 client.16215 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24c17fa640 1 -- 172.21.15.107:0/1976380060 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcdd4e7bb0 con 0x55fcdd5904e0 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24c17fa640 10 client.16215 handle_command_reply: tid=0 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 client.16215 shutdown 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 msgr2=0x55fcdd58bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f24c8031440 tx=0x7f24c80315b0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 shutdown_connections 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5904e0 0x55fcdd592950 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58bcb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24c17fa640 0 client.16215 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57b030 0x55fcdd584c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/1976380060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57a640 0x55fcdd584730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 >> 172.21.15.107:0/1976380060 conn(0x55fcdd5718a0 msgr2=0x55fcdd570b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 shutdown_connections 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 1 -- 172.21.15.107:0/1976380060 wait complete. 2024-04-24T23:47:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.442+0000 7f24e2c4c740 20 client.16215 trim_cache size 0 max 16384 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.455+0000 7f24e2c4c740 1 Processor -- start 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.455+0000 7f24e2c4c740 1 -- start start 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd58bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd5747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a8350 con 0x55fcdd57a640 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a9e90 con 0x55fcdd57b030 2024-04-24T23:47:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a8a20 con 0x55fcdd5801a0 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43532/0 (socket says 172.21.15.107:43532) 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 -- 172.21.15.107:0/3231809267 learned_addr learned my addr 172.21.15.107:0/3231809267 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24ddce4640 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd58bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 -- 172.21.15.107:0/3231809267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 msgr2=0x55fcdd5747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd5747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 -- 172.21.15.107:0/3231809267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 msgr2=0x55fcdd58bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd58bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 -- 172.21.15.107:0/3231809267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdd2b87d0 con 0x55fcdd57a640 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24ddce4640 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd58bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f24c802a930 tx=0x7f24c8032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2beb1853ffdc1b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24c37fe640 1 -- 172.21.15.107:0/3231809267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035040 con 0x55fcdd57a640 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24c37fe640 1 -- 172.21.15.107:0/3231809267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c8032dd0 con 0x55fcdd57a640 2024-04-24T23:47:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.456+0000 7f24c37fe640 1 -- 172.21.15.107:0/3231809267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c80049d0 con 0x55fcdd57a640 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- 172.21.15.107:0/3231809267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 msgr2=0x55fcdd5709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f24c802a930 tx=0x7f24c8032960 comp rx=0 tx=0).stop 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- 172.21.15.107:0/3231809267 shutdown_connections 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd5801a0 0x55fcdd5747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd58bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/3231809267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd57a640 0x55fcdd5709c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- 172.21.15.107:0/3231809267 >> 172.21.15.107:0/3231809267 conn(0x55fcdd5718a0 msgr2=0x55fcdd5702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- 172.21.15.107:0/3231809267 shutdown_connections 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- 172.21.15.107:0/3231809267 wait complete. 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- start start 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 Processor -- start 2024-04-24T23:47:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57a640 0x55fcdd588f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd589440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a97c0 con 0x55fcdd5801a0 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a90f0 con 0x55fcdd57b030 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.457+0000 7f24e2c4c740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a6330 con 0x55fcdd57a640 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24e2c4c740 10 client.0 _ll_register_callbacks cb 0x7f24c8032a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43546/0 (socket says 172.21.15.107:43546) 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 -- 172.21.15.107:0/2952240296 learned_addr learned my addr 172.21.15.107:0/2952240296 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57a640 0x55fcdd588f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd589440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 -- 172.21.15.107:0/2952240296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57a640 msgr2=0x55fcdd588f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57a640 0x55fcdd588f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 -- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 msgr2=0x55fcdd589440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd589440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 -- 172.21.15.107:0/2952240296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdd58aa40 con 0x55fcdd5801a0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24dd4e3640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f24c8032b90 tx=0x7f24c80041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd83fe8293680ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035070 con 0x55fcdd5801a0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f24c8032250 con 0x55fcdd5801a0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24c8032dd0 con 0x55fcdd5801a0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24c8065050 con 0x55fcdd5801a0 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24e2c4c740 10 client.16221 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcdd58bd40 con 0x55fcdd5801a0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.458+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c8032dd0 con 0x55fcdd5801a0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24c80316b0 con 0x55fcdd5801a0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 10 client.16221 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 10 client.16221 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 20 client.16221 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5f6f70 0x55fcdd5f9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 4 client.16221 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcdd4e1f00 con 0x55fcdd5f6f70 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5f6f70 0x55fcdd5f9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24c17fa640 10 client.16221 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24ddce4640 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5f6f70 0x55fcdd5f9360 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e5d926908fa6aed server_cookie=3e065759ac8910f0 in_seq=0 out_seq=0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24c17fa640 1 -- 172.21.15.107:0/2952240296 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcdd4e1f00 con 0x55fcdd5f6f70 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24c17fa640 10 client.16221 handle_command_reply: tid=0 2024-04-24T23:47:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 1 client.16221 shutdown 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 msgr2=0x55fcdd58a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.459+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f24c8032b90 tx=0x7f24c80041a0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 shutdown_connections 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x55fcdd5f6f70 0x55fcdd5f9360 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcdd5801a0 0x55fcdd58a450 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24c17fa640 0 client.16221 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fcdd57b030 0x55fcdd589440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 --2- 172.21.15.107:0/2952240296 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55fcdd57a640 0x55fcdd588f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 >> 172.21.15.107:0/2952240296 conn(0x55fcdd5718a0 msgr2=0x55fcdd570b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 shutdown_connections 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 1 -- 172.21.15.107:0/2952240296 wait complete. 2024-04-24T23:47:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24e2c4c740 20 client.16221 trim_cache size 0 max 16384 2024-04-24T23:47:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:47:32.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 msgr2=0x7f24c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 0x7f24c40a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24d009b660 tx=0x7f24d006b810 comp rx=0 tx=0).stop 2024-04-24T23:47:32.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 msgr2=0x7f24d819ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f24d000cad0 tx=0x7f24d000cf90 comp rx=0 tx=0).stop 2024-04-24T23:47:32.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 shutdown_connections 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f24c40a2b90 0x7f24c40a5050 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24d8102230 0x7f24d819ee70 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24d8101870 0x7f24d819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 --2- 172.21.15.107:0/1383998469 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f24d8100680 0x7f24d80694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 >> 172.21.15.107:0/1383998469 conn(0x7f24d80fbbf0 msgr2=0x7f24d8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 shutdown_connections 2024-04-24T23:47:32.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.460+0000 7f24dff48640 1 -- 172.21.15.107:0/1383998469 wait complete. 2024-04-24T23:47:32.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T23:47:32.492 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:32 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/702881759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:47:32.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:32 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/702881759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:47:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 -- 172.21.15.107:0/3135170612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c102200 msgr2=0x7f333c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 --2- 172.21.15.107:0/3135170612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c102200 0x7f333c1045f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f332c009900 tx=0x7f332c02a410 comp rx=0 tx=0).stop 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 -- 172.21.15.107:0/3135170612 shutdown_connections 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 --2- 172.21.15.107:0/3135170612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c107460 0x7f333c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 --2- 172.21.15.107:0/3135170612 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c104b30 0x7f333c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 --2- 172.21.15.107:0/3135170612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c102200 0x7f333c1045f0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 -- 172.21.15.107:0/3135170612 >> 172.21.15.107:0/3135170612 conn(0x7f333c0fbbf0 msgr2=0x7f333c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 -- 172.21.15.107:0/3135170612 shutdown_connections 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.571+0000 7f3342542640 1 -- 172.21.15.107:0/3135170612 wait complete. 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 Processor -- start 2024-04-24T23:47:32.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 -- start start 2024-04-24T23:47:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c102200 0x7f333c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c107460 0x7f333c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f333c10d400 con 0x7f333c104b30 2024-04-24T23:47:32.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f333c10d280 con 0x7f333c102200 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f3342542640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f333c10d580 con 0x7f333c107460 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43564/0 (socket says 172.21.15.107:43564) 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.572+0000 7f333b7fe640 1 -- 172.21.15.107:0/2372971499 learned_addr learned my addr 172.21.15.107:0/2372971499 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333bfff640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c102200 0x7f333c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f3340adf640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c107460 0x7f333c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 -- 172.21.15.107:0/2372971499 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c107460 msgr2=0x7f333c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c107460 0x7f333c19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 -- 172.21.15.107:0/2372971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c102200 msgr2=0x7f333c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c102200 0x7f333c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 -- 172.21.15.107:0/2372971499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c19f790 con 0x7f333c104b30 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f333b7fe640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f333000d770 tx=0x7f333000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f33397fa640 1 -- 172.21.15.107:0/2372971499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330017070 con 0x7f333c104b30 2024-04-24T23:47:32.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f33397fa640 1 -- 172.21.15.107:0/2372971499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3330009c30 con 0x7f333c104b30 2024-04-24T23:47:32.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333c19fa80 con 0x7f333c104b30 2024-04-24T23:47:32.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f33397fa640 1 -- 172.21.15.107:0/2372971499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333000f620 con 0x7f333c104b30 2024-04-24T23:47:32.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.573+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f333c19ffd0 con 0x7f333c104b30 2024-04-24T23:47:32.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.575+0000 7f33397fa640 1 -- 172.21.15.107:0/2372971499 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f333001a070 con 0x7f333c104b30 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.575+0000 7f3345252740 1 Processor -- start 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.575+0000 7f3345252740 1 -- start start 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3480720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f3483220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5563f33df670 con 0x5563f3480d90 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5563f3387970 con 0x5563f347b230 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3345252740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5563f33db630 con 0x5563f347bc20 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f3483220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:44000/0 (socket says 172.21.15.107:44000) 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 -- 172.21.15.107:0/1970142184 learned_addr learned my addr 172.21.15.107:0/1970142184 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f333bfff640 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3480720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 -- 172.21.15.107:0/1970142184 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 msgr2=0x5563f3480720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3480720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 -- 172.21.15.107:0/1970142184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 msgr2=0x5563f3483220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f3483220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 -- 172.21.15.107:0/1970142184 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f33ab400 con 0x5563f347b230 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f333b7fe640 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f3483220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f33397fa640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 0x7f33240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f333bfff640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 0x7f33240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f3340adf640 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e90 tx=0x7f333400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac798e8f0f192071 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f33397fa640 1 -- 172.21.15.107:0/2372971499 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f33300cef20 con 0x7f333c104b30 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f33227fc640 1 -- 172.21.15.107:0/1970142184 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334004d20 con 0x5563f347b230 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f33227fc640 1 -- 172.21.15.107:0/1970142184 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3334005500 con 0x5563f347b230 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.576+0000 7f33227fc640 1 -- 172.21.15.107:0/1970142184 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33340118e0 con 0x5563f347b230 2024-04-24T23:47:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- 172.21.15.107:0/1970142184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 msgr2=0x5563f347b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f333bfff640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 0x7f33240a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f332c0314e0 tx=0x7f332c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e90 tx=0x7f333400f730 comp rx=0 tx=0).stop 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- 172.21.15.107:0/1970142184 shutdown_connections 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f3483220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3480720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 --2- 172.21.15.107:0/1970142184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347b230 0x5563f347b650 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- 172.21.15.107:0/1970142184 >> 172.21.15.107:0/1970142184 conn(0x5563f34768a0 msgr2=0x5563f3476ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- 172.21.15.107:0/1970142184 shutdown_connections 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- 172.21.15.107:0/1970142184 wait complete. 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.577+0000 7f3345252740 1 -- start start 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 Processor -- start 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3485840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f348c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5563f3483d40 con 0x5563f347b230 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5563f339b050 con 0x5563f3480d90 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5563f3483ec0 con 0x5563f347bc20 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3345252740 10 client.0 _ll_register_callbacks cb 0x5563f30f2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43586/0 (socket says 172.21.15.107:43586) 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 -- 172.21.15.107:0/3053956217 learned_addr learned my addr 172.21.15.107:0/3053956217 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f3340adf640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f348c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333b7fe640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3485840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 -- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 msgr2=0x5563f3485840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3485840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 -- 172.21.15.107:0/3053956217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 msgr2=0x5563f348c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f348c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 -- 172.21.15.107:0/3053956217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f336cdc0 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f333bfff640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f332c02a930 tx=0x7f332c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f937d7399dbee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f331b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c04d030 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.578+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f332c005450 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3334005500 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f332c05a050 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 10 client.16233 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563f30f33f0 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3334005500 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f332c03d890 con 0x5563f347b230 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 10 client.16233 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 10 client.16233 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:47:32.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 20 client.16233 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f3491120 0x5563f3493570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 4 client.16233 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563f33ecbb0 con 0x5563f3491120 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.579+0000 7f333b7fe640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f3491120 0x5563f3493570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.580+0000 7f333b7fe640 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f3491120 0x5563f3493570 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=795016170c7941d server_cookie=b2b556ef73ea743b in_seq=0 out_seq=0 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.580+0000 7f331b7fe640 10 client.16233 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.580+0000 7f331b7fe640 1 -- 172.21.15.107:0/3053956217 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563f33ecbb0 con 0x5563f3491120 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.580+0000 7f331b7fe640 10 client.16233 handle_command_reply: tid=0 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.580+0000 7f3345252740 1 client.16233 shutdown 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 msgr2=0x5563f3485300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f332c02a930 tx=0x7f332c009260 comp rx=0 tx=0).stop 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 shutdown_connections 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f3491120 0x5563f3493570 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f348c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f331b7fe640 0 client.16233 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347bc20 0x5563f3485840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 --2- 172.21.15.107:0/3053956217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347b230 0x5563f3485300 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 >> 172.21.15.107:0/3053956217 conn(0x5563f34768a0 msgr2=0x5563f3475b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 shutdown_connections 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 1 -- 172.21.15.107:0/3053956217 wait complete. 2024-04-24T23:47:32.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.581+0000 7f3345252740 20 client.16233 trim_cache size 0 max 16384 2024-04-24T23:47:32.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:32 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/702881759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:47:32.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 Processor -- start 2024-04-24T23:47:32.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 -- start start 2024-04-24T23:47:32.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f3479580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f3479ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1aa5e0 con 0x5563f347bc20 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1ac120 con 0x5563f3480d90 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3345252740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1aacb0 con 0x5563f347b230 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43598/0 (socket says 172.21.15.107:43598) 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 -- 172.21.15.107:0/57064133 learned_addr learned my addr 172.21.15.107:0/57064133 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3340adf640 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f3479ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 -- 172.21.15.107:0/57064133 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 msgr2=0x5563f3479580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f3479580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 -- 172.21.15.107:0/57064133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 msgr2=0x5563f3479ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f3479ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 -- 172.21.15.107:0/57064133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f31bd7d0 con 0x5563f347bc20 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f333b7fe640 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f333009b340 tx=0x7f3330010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18b88c879578ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.594+0000 7f3321ffb640 1 -- 172.21.15.107:0/57064133 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330021040 con 0x5563f347bc20 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3321ffb640 1 -- 172.21.15.107:0/57064133 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f333009bab0 con 0x5563f347bc20 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3321ffb640 1 -- 172.21.15.107:0/57064133 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330099740 con 0x5563f347bc20 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- 172.21.15.107:0/57064133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 msgr2=0x5563f348c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f333009b340 tx=0x7f3330010cd0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- 172.21.15.107:0/57064133 shutdown_connections 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f3480d90 0x5563f3479ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f347bc20 0x5563f348c840 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 --2- 172.21.15.107:0/57064133 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f3479580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- 172.21.15.107:0/57064133 >> 172.21.15.107:0/57064133 conn(0x5563f34768a0 msgr2=0x5563f3475b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- 172.21.15.107:0/57064133 shutdown_connections 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- 172.21.15.107:0/57064133 wait complete. 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.595+0000 7f3345252740 1 -- start start 2024-04-24T23:47:32.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 Processor -- start 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f348a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347bc20 0x5563f348a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1aba50 con 0x5563f3480d90 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1ab380 con 0x5563f347bc20 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1a85c0 con 0x5563f347b230 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3345252740 10 client.0 _ll_register_callbacks cb 0x7f332c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43610/0 (socket says 172.21.15.107:43610) 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 -- 172.21.15.107:0/1767848525 learned_addr learned my addr 172.21.15.107:0/1767848525 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f333bfff640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f348a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f333b7fe640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347bc20 0x5563f348a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 -- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 msgr2=0x5563f348a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f348a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 -- 172.21.15.107:0/1767848525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347bc20 msgr2=0x5563f348a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347bc20 0x5563f348a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 -- 172.21.15.107:0/1767848525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f348bd90 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f3340adf640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e60 tx=0x7f33340049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39664811eea79d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.596+0000 7f331b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333402b060 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3334011640 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f333009bab0 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3334032050 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 10 client.16239 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563f34f79c0 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f333009bab0 con 0x5563f3480d90 2024-04-24T23:47:32.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3334025070 con 0x5563f3480d90 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 10 client.16239 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 10 client.16239 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 20 client.16239 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f34f8130 0x5563f34fa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 4 client.16239 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.597+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563f33e6f00 con 0x5563f34f8130 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f333bfff640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f34f8130 0x5563f34fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f333bfff640 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f34f8130 0x5563f34fa550 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7410b58c19e9be4 server_cookie=513b40314cfb3cbb in_seq=0 out_seq=0 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f331b7fe640 10 client.16239 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f331b7fe640 1 -- 172.21.15.107:0/1767848525 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563f33e6f00 con 0x5563f34f8130 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f331b7fe640 10 client.16239 handle_command_reply: tid=0 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 client.16239 shutdown 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 msgr2=0x5563f348b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e60 tx=0x7f33340049e0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 shutdown_connections 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563f3480d90 0x5563f348b7a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x5563f34f8130 0x5563f34fa550 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563f347bc20 0x5563f348a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f331b7fe640 0 client.16239 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 --2- 172.21.15.107:0/1767848525 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5563f347b230 0x5563f348a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 >> 172.21.15.107:0/1767848525 conn(0x5563f34768a0 msgr2=0x5563f347c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 shutdown_connections 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 1 -- 172.21.15.107:0/1767848525 wait complete. 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.598+0000 7f3345252740 20 client.16239 trim_cache size 0 max 16384 2024-04-24T23:47:32.605 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:47:32.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 msgr2=0x7f33240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 0x7f33240a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f332c0314e0 tx=0x7f332c005fd0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 msgr2=0x7f333c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f333000d770 tx=0x7f333000dc30 comp rx=0 tx=0).stop 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 shutdown_connections 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f33240a2b90 0x7f33240a5050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f333c107460 0x7f333c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c104b30 0x7f333c1980b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 --2- 172.21.15.107:0/2372971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f333c102200 0x7f333c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 >> 172.21.15.107:0/2372971499 conn(0x7f333c0fbbf0 msgr2=0x7f333c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 shutdown_connections 2024-04-24T23:47:32.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.599+0000 7f3342542640 1 -- 172.21.15.107:0/2372971499 wait complete. 2024-04-24T23:47:32.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T23:47:32.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 -- 172.21.15.107:0/177176721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 msgr2=0x7f79e8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 --2- 172.21.15.107:0/177176721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8100aa0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009ca0 tx=0x7f79d802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 -- 172.21.15.107:0/177176721 shutdown_connections 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 --2- 172.21.15.107:0/177176721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 0x7f79e8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 --2- 172.21.15.107:0/177176721 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 --2- 172.21.15.107:0/177176721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8100aa0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 -- 172.21.15.107:0/177176721 >> 172.21.15.107:0/177176721 conn(0x7f79e80fbbf0 msgr2=0x7f79e80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 -- 172.21.15.107:0/177176721 shutdown_connections 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.749+0000 7f79ecd14640 1 -- 172.21.15.107:0/177176721 wait complete. 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 Processor -- start 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 -- start start 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 0x7f79e819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810c300 con 0x7f79e8100680 2024-04-24T23:47:32.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810c180 con 0x7f79e8102230 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79ecd14640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810c480 con 0x7f79e8101870 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79e659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79e659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43620/0 (socket says 172.21.15.107:43620) 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79e659c640 1 -- 172.21.15.107:0/1290764782 learned_addr learned my addr 172.21.15.107:0/1290764782 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.750+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 0x7f79e819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 -- 172.21.15.107:0/1290764782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 msgr2=0x7f79e8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 -- 172.21.15.107:0/1290764782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 msgr2=0x7f79e819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 0x7f79e819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 -- 172.21.15.107:0/1290764782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e819f700 con 0x7f79e8100680 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79e659c640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f79d8004af0 tx=0x7f79d8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79d77fe640 1 -- 172.21.15.107:0/1290764782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d802fb60 con 0x7f79e8100680 2024-04-24T23:47:32.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79d77fe640 1 -- 172.21.15.107:0/1290764782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d802fd00 con 0x7f79e8100680 2024-04-24T23:47:32.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e819f990 con 0x7f79e8100680 2024-04-24T23:47:32.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79d77fe640 1 -- 172.21.15.107:0/1290764782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8046920 con 0x7f79e8100680 2024-04-24T23:47:32.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.751+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e819fee0 con 0x7f79e8100680 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.753+0000 7f79d77fe640 1 -- 172.21.15.107:0/1290764782 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f79d8030c30 con 0x7f79e8100680 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 Processor -- start 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 -- start start 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df661720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a8df5c0670 con 0x55a8df65c230 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a8df568970 con 0x55a8df661d90 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79efa25740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55a8df5bc630 con 0x55a8df65cc20 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79d77fe640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 0x7f79d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:44060/0 (socket says 172.21.15.107:44060) 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43630/0 (socket says 172.21.15.107:43630) 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e5d9b640 1 -- 172.21.15.107:0/2567513097 learned_addr learned my addr 172.21.15.107:0/2567513097 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 0x7f79d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79d77fe640 1 -- 172.21.15.107:0/1290764782 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f79d80f0a50 con 0x7f79e8100680 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.754+0000 7f79e659c640 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df661720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 -- 172.21.15.107:0/2567513097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 msgr2=0x55a8df661720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df661720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 -- 172.21.15.107:0/2567513097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 msgr2=0x55a8df65c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 -- 172.21.15.107:0/2567513097 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df58c400 con 0x55a8df661d90 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 0x7f79d00a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55a8df65d480 tx=0x7f79e0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009c20 tx=0x7f79dc00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5082c82b323d369c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79d4ff9640 1 -- 172.21.15.107:0/2567513097 <== mon.2 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc016070 con 0x55a8df661d90 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79d4ff9640 1 -- 172.21.15.107:0/2567513097 <== mon.2 v2:172.21.15.170:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79dc011950 con 0x55a8df661d90 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.755+0000 7f79d4ff9640 1 -- 172.21.15.107:0/2567513097 <== mon.2 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc010930 con 0x55a8df661d90 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- 172.21.15.107:0/2567513097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 msgr2=0x55a8df664220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009c20 tx=0x7f79dc00e6c0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- 172.21.15.107:0/2567513097 shutdown_connections 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df664220 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df661720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- 172.21.15.107:0/2567513097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65c230 0x55a8df65c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- 172.21.15.107:0/2567513097 >> 172.21.15.107:0/2567513097 conn(0x55a8df6578a0 msgr2=0x55a8df657ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- 172.21.15.107:0/2567513097 shutdown_connections 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- 172.21.15.107:0/2567513097 wait complete. 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- start start 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 Processor -- start 2024-04-24T23:47:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65c230 0x55a8df6662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df66d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a8df664d40 con 0x55a8df65cc20 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a8df57c050 con 0x55a8df661d90 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55a8df664ec0 con 0x55a8df65c230 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79efa25740 10 client.0 _ll_register_callbacks cb 0x55a8df2d3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43644/0 (socket says 172.21.15.107:43644) 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.756+0000 7f79e5d9b640 1 -- 172.21.15.107:0/817290381 learned_addr learned my addr 172.21.15.107:0/817290381 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df66d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e659c640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65c230 0x55a8df6662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 -- 172.21.15.107:0/817290381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65c230 msgr2=0x55a8df6662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65c230 0x55a8df6662a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 -- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 msgr2=0x55a8df66d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df66d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 -- 172.21.15.107:0/817290381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df54ddc0 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009f40 tx=0x7f79dc011830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7351a0ac9ad275e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc016050 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f79dc005040 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79dc011950 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79dc005240 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79efa25740 10 client.16251 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8df2d43f0 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79dc011950 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.757+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f79dc02b030 con 0x55a8df65cc20 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 10 client.16251 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:32.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 20 client.16251 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df671e80 0x55a8df6742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 4 client.16251 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8df5cdbb0 con 0x55a8df671e80 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79e659c640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df671e80 0x55a8df6742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79cdffb640 10 client.16251 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.758+0000 7f79e659c640 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df671e80 0x55a8df6742e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5498a30e8fb7a40 server_cookie=66344ca94e5a54ee in_seq=0 out_seq=0 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79cdffb640 1 -- 172.21.15.107:0/817290381 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8df5cdbb0 con 0x55a8df671e80 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79cdffb640 10 client.16251 handle_command_reply: tid=0 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 client.16251 shutdown 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 msgr2=0x55a8df6667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009f40 tx=0x7f79dc011830 comp rx=0 tx=0).stop 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 shutdown_connections 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df671e80 0x55a8df6742e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df661d90 0x55a8df66d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df65cc20 0x55a8df6667e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 --2- 172.21.15.107:0/817290381 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65c230 0x55a8df6662a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79cdffb640 0 client.16251 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 >> 172.21.15.107:0/817290381 conn(0x55a8df6578a0 msgr2=0x55a8df656b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 shutdown_connections 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 1 -- 172.21.15.107:0/817290381 wait complete. 2024-04-24T23:47:32.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.759+0000 7f79efa25740 20 client.16251 trim_cache size 0 max 16384 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 Processor -- start 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 -- start start 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df65a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a8290 con 0x55a8df661d90 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a9dd0 con 0x55a8df65c230 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.772+0000 7f79efa25740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a8960 con 0x55a8df65cc20 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43650/0 (socket says 172.21.15.107:43650) 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 -- 172.21.15.107:0/1851041924 learned_addr learned my addr 172.21.15.107:0/1851041924 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 -- 172.21.15.107:0/1851041924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 msgr2=0x55a8df66d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e659c640 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df65a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 -- 172.21.15.107:0/1851041924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 msgr2=0x55a8df65a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df65a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 -- 172.21.15.107:0/1851041924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df39e7d0 con 0x55a8df661d90 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e659c640 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df65a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f79e000c400 tx=0x7f79e0013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11340fa29d40ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79cffff640 1 -- 172.21.15.107:0/1851041924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e000cdd0 con 0x55a8df661d90 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79cffff640 1 -- 172.21.15.107:0/1851041924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0005500 con 0x55a8df661d90 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.773+0000 7f79cffff640 1 -- 172.21.15.107:0/1851041924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00119b0 con 0x55a8df661d90 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- 172.21.15.107:0/1851041924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 msgr2=0x55a8df65acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f79e000c400 tx=0x7f79e0013830 comp rx=0 tx=0).stop 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- 172.21.15.107:0/1851041924 shutdown_connections 2024-04-24T23:47:32.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df65acc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- 172.21.15.107:0/1851041924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df65a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- 172.21.15.107:0/1851041924 >> 172.21.15.107:0/1851041924 conn(0x55a8df6578a0 msgr2=0x55a8df656b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- 172.21.15.107:0/1851041924 shutdown_connections 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- 172.21.15.107:0/1851041924 wait complete. 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- start start 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 Processor -- start 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df66b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a9700 con 0x55a8df661d90 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a9030 con 0x55a8df65c230 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81a6270 con 0x55a8df65cc20 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79efa25740 10 client.0 _ll_register_callbacks cb 0x7f79e0013f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43656/0 (socket says 172.21.15.107:43656) 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79e6d9d640 1 -- 172.21.15.107:0/4267013659 learned_addr learned my addr 172.21.15.107:0/4267013659 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.774+0000 7f79e659c640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df66b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 -- 172.21.15.107:0/4267013659 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 msgr2=0x55a8df66b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e5d9b640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 -- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 msgr2=0x55a8df66b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df66b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 -- 172.21.15.107:0/4267013659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df66cb10 con 0x55a8df661d90 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79e6d9d640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f79e002d040 tx=0x7f79e0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f90df2ea217adbf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0004ee0 con 0x55a8df661d90 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f79e0005080 con 0x55a8df661d90 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e0005500 con 0x55a8df661d90 2024-04-24T23:47:32.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79e0005280 con 0x55a8df661d90 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79efa25740 10 client.16257 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.775+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8df6d84c0 con 0x55a8df661d90 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0005500 con 0x55a8df661d90 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f79e0034070 con 0x55a8df661d90 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 10 client.16257 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 20 client.16257 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df6d8e90 0x55a8df6db2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 4 client.16257 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.776+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8df666090 con 0x55a8df6d8e90 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79e659c640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df6d8e90 0x55a8df6db2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79cdffb640 10 client.16257 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:32.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79e659c640 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df6d8e90 0x55a8df6db2b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e78e0e50a0e74d server_cookie=f1aa99a2a7f25a7f in_seq=0 out_seq=0 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79cdffb640 1 -- 172.21.15.107:0/4267013659 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8df666090 con 0x55a8df6d8e90 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79cdffb640 10 client.16257 handle_command_reply: tid=0 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 client.16257 shutdown 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 msgr2=0x55a8df66c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f79e002d040 tx=0x7f79e0005740 comp rx=0 tx=0).stop 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 shutdown_connections 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55a8df6d8e90 0x55a8df6db2b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8df661d90 0x55a8df66c520 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79cdffb640 0 client.16257 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55a8df65cc20 0x55a8df66b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 --2- 172.21.15.107:0/4267013659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a8df65c230 0x55a8df66b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 >> 172.21.15.107:0/4267013659 conn(0x55a8df6578a0 msgr2=0x55a8df65d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 shutdown_connections 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 1 -- 172.21.15.107:0/4267013659 wait complete. 2024-04-24T23:47:32.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.777+0000 7f79efa25740 20 client.16257 trim_cache size 0 max 16384 2024-04-24T23:47:32.784 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 msgr2=0x7f79d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 0x7f79d00a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55a8df65d480 tx=0x7f79e0005850 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 msgr2=0x7f79e8197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f79d8004af0 tx=0x7f79d8031ee0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 shutdown_connections 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f79d00a2b90 0x7f79d00a5050 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e8102230 0x7f79e819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f79e8101870 0x7f79e8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 --2- 172.21.15.107:0/1290764782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79e8100680 0x7f79e8197ae0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 >> 172.21.15.107:0/1290764782 conn(0x7f79e80fbbf0 msgr2=0x7f79e80688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 shutdown_connections 2024-04-24T23:47:32.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:32.778+0000 7f79ecd14640 1 -- 172.21.15.107:0/1290764782 wait complete. 2024-04-24T23:47:33.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:33 smithi195 ceph-mon[33019]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 586 MiB/s wr, 284 op/s 2024-04-24T23:47:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:33 smithi170 ceph-mon[32714]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 586 MiB/s wr, 284 op/s 2024-04-24T23:47:33.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:33 smithi107 ceph-mon[29118]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 586 MiB/s wr, 284 op/s 2024-04-24T23:47:33.789 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T23:47:33.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:47:33.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T23:47:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 -- 172.21.15.107:0/2074364857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 msgr2=0x7f10e4109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/2074364857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e4109bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009ca0 tx=0x7f10dc02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 -- 172.21.15.107:0/2074364857 shutdown_connections 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/2074364857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e4109bd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/2074364857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4102980 0x7f10e4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/2074364857 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4101790 0x7f10e4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.902+0000 7f10e9e5d640 1 -- 172.21.15.107:0/2074364857 >> 172.21.15.107:0/2074364857 conn(0x7f10e40fcd20 msgr2=0x7f10e40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- 172.21.15.107:0/2074364857 shutdown_connections 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- 172.21.15.107:0/2074364857 wait complete. 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 Processor -- start 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- start start 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4101790 0x7f10e4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 0x7f10e4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f10e410d410 con 0x7f10e4103340 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f10e410d290 con 0x7f10e4101790 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.903+0000 7f10e9e5d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f10e410d590 con 0x7f10e4102980 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43670/0 (socket says 172.21.15.107:43670) 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 -- 172.21.15.107:0/1057191519 learned_addr learned my addr 172.21.15.107:0/1057191519 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e37fe640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4101790 0x7f10e4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 0x7f10e4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 -- 172.21.15.107:0/1057191519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 msgr2=0x7f10e4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 0x7f10e4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 -- 172.21.15.107:0/1057191519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4101790 msgr2=0x7f10e4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4101790 0x7f10e4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 -- 172.21.15.107:0/1057191519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e419f670 con 0x7f10e4103340 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 0x7f10e4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005e30 tx=0x7f10dc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e0ff9640 1 -- 172.21.15.107:0/1057191519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc003c30 con 0x7f10e4103340 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e419f900 con 0x7f10e4103340 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e0ff9640 1 -- 172.21.15.107:0/1057191519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10dc003dd0 con 0x7f10e4103340 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e0ff9640 1 -- 172.21.15.107:0/1057191519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc02fde0 con 0x7f10e4103340 2024-04-24T23:47:33.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.904+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e419fe50 con 0x7f10e4103340 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.906+0000 7f10e0ff9640 1 -- 172.21.15.107:0/1057191519 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f10dc004050 con 0x7f10e4103340 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e0ff9640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 0x7f10cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 Processor -- start 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e37fe640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 0x7f10cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e0ff9640 1 -- 172.21.15.107:0/1057191519 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f10dc0f09d0 con 0x7f10e4103340 2024-04-24T23:47:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 -- start start 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c0e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c13690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563241b6d2c0 con 0x563241c13d00 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563241b15910 con 0x563241c0eb90 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x563241b69280 con 0x563241c0e1a0 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43672/0 (socket says 172.21.15.107:43672) 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e37fe640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 0x7f10cc0a5050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f10d4005fd0 tx=0x7f10d4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e3fff640 1 -- 172.21.15.107:0/1960477621 learned_addr learned my addr 172.21.15.107:0/1960477621 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.907+0000 7f10e37fe640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c0e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c13690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 -- 172.21.15.107:0/1960477621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 msgr2=0x563241c0e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c0e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 -- 172.21.15.107:0/1960477621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 msgr2=0x563241c13690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c13690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 -- 172.21.15.107:0/1960477621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563241b26fb0 con 0x563241c13d00 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c13690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10e3fff640 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b93e0 tx=0x7f10dc02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7053e639673bc2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10c9ffb640 1 -- 172.21.15.107:0/1960477621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0bcf80 con 0x563241c13d00 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10c9ffb640 1 -- 172.21.15.107:0/1960477621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10dc0d7070 con 0x563241c13d00 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.908+0000 7f10c9ffb640 1 -- 172.21.15.107:0/1960477621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc004340 con 0x563241c13d00 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- 172.21.15.107:0/1960477621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 msgr2=0x563241c16190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b93e0 tx=0x7f10dc02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- 172.21.15.107:0/1960477621 shutdown_connections 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c16190 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c13690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/1960477621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c0e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- 172.21.15.107:0/1960477621 >> 172.21.15.107:0/1960477621 conn(0x563241bd1950 msgr2=0x563241c05810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- 172.21.15.107:0/1960477621 shutdown_connections 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- 172.21.15.107:0/1960477621 wait complete. 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- start start 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 Processor -- start 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c18160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c13d00 0x563241c1f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.909+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563241b6d2c0 con 0x563241c0eb90 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563241b28e90 con 0x563241c13d00 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x563241c16a70 con 0x563241c0e1a0 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 10 client.0 _ll_register_callbacks cb 0x563241b64970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43676/0 (socket says 172.21.15.107:43676) 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 -- 172.21.15.107:0/618716022 learned_addr learned my addr 172.21.15.107:0/618716022 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e37fe640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c18160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e3fff640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c13d00 0x563241c1f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 -- 172.21.15.107:0/618716022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 msgr2=0x563241c18160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c18160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 -- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c13d00 msgr2=0x563241c1f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c13d00 0x563241c1f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 -- 172.21.15.107:0/618716022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632419497e0 con 0x563241c0eb90 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f10d800de60 tx=0x7f10d800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661e5ffb0fe6e08c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d800fea0 con 0x563241c0eb90 2024-04-24T23:47:33.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f10d8004220 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10dc0d7070 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10d8004420 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 10 client.16269 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.910+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632419a9f70 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10dc0d7070 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f10d8003ae0 con 0x563241c0eb90 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 10 client.16269 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 20 client.16269 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c23d80 0x563241c26200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 4 client.16269 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563241b415a0 con 0x563241c23d80 2024-04-24T23:47:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.911+0000 7f10e37fe640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c23d80 0x563241c26200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.912+0000 7f10c2ffd640 10 client.16269 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.912+0000 7f10e37fe640 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c23d80 0x563241c26200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ffe2200eb27ee server_cookie=36619eb6ac7040fb in_seq=0 out_seq=0 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.912+0000 7f10c2ffd640 1 -- 172.21.15.107:0/618716022 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563241b415a0 con 0x563241c23d80 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.912+0000 7f10c2ffd640 10 client.16269 handle_command_reply: tid=0 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.912+0000 7f10ecb6f740 1 client.16269 shutdown 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 msgr2=0x563241c186a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f10d800de60 tx=0x7f10d800f730 comp rx=0 tx=0).stop 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 shutdown_connections 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c23d80 0x563241c26200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c13d00 0x563241c1f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0eb90 0x563241c186a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10c2ffd640 0 client.16269 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/618716022 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0e1a0 0x563241c18160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 >> 172.21.15.107:0/618716022 conn(0x563241bd1950 msgr2=0x563241c04950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 shutdown_connections 2024-04-24T23:47:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 1 -- 172.21.15.107:0/618716022 wait complete. 2024-04-24T23:47:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.913+0000 7f10ecb6f740 20 client.16269 trim_cache size 0 max 16384 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.930+0000 7f10ecb6f740 1 Processor -- start 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.930+0000 7f10ecb6f740 1 -- start start 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c1f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c13d00 0x563241c04600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a82d0 con 0x563241c0e1a0 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a9e10 con 0x563241c0eb90 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a89a0 con 0x563241c13d00 2024-04-24T23:47:33.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43680/0 (socket says 172.21.15.107:43680) 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 -- 172.21.15.107:0/3520431422 learned_addr learned my addr 172.21.15.107:0/3520431422 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 -- 172.21.15.107:0/3520431422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c13d00 msgr2=0x563241c04600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c13d00 0x563241c04600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 -- 172.21.15.107:0/3520431422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 msgr2=0x563241c1f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c1f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c1f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 -- 172.21.15.107:0/3520431422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563241841170 con 0x563241c0e1a0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c1f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10e37fe640 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f10d40058c0 tx=0x7f10d4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f1f732b771a5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10c97fa640 1 -- 172.21.15.107:0/3520431422 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4011c70 con 0x563241c0e1a0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10c97fa640 1 -- 172.21.15.107:0/3520431422 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10d4011e10 con 0x563241c0e1a0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.931+0000 7f10c97fa640 1 -- 172.21.15.107:0/3520431422 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d402b7d0 con 0x563241c0e1a0 2024-04-24T23:47:33.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- 172.21.15.107:0/3520431422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 msgr2=0x563241c040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f10d40058c0 tx=0x7f10d4012000 comp rx=0 tx=0).stop 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- 172.21.15.107:0/3520431422 shutdown_connections 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c13d00 0x563241c04600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0eb90 0x563241c1f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/3520431422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c0e1a0 0x563241c040c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- 172.21.15.107:0/3520431422 >> 172.21.15.107:0/3520431422 conn(0x563241bd1950 msgr2=0x563241c0ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- 172.21.15.107:0/3520431422 shutdown_connections 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- 172.21.15.107:0/3520431422 wait complete. 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 -- start start 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.932+0000 7f10ecb6f740 1 Processor -- start 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 0x563241c1d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0eb90 0x563241c1dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a9740 con 0x563241c13d00 2024-04-24T23:47:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a9070 con 0x563241c0e1a0 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41a62b0 con 0x563241c0eb90 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 10 client.0 _ll_register_callbacks cb 0x7f10d800d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43682/0 (socket says 172.21.15.107:43682) 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 -- 172.21.15.107:0/660038709 learned_addr learned my addr 172.21.15.107:0/660038709 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e2ffd640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0eb90 0x563241c1dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e37fe640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 0x563241c1d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 -- 172.21.15.107:0/660038709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0eb90 msgr2=0x563241c1dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0eb90 0x563241c1dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 -- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 msgr2=0x563241c1d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 0x563241c1d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 -- 172.21.15.107:0/660038709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563241c1f1a0 con 0x563241c13d00 2024-04-24T23:47:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e37fe640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 0x563241c1d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10e3fff640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc02f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3a11c9cbe3def2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc087060 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f10dc004340 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10d4011e10 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10dc0bb020 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.933+0000 7f10ecb6f740 10 client.16275 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563241c8aaf0 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10d4011e10 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f10dc0bc3d0 con 0x563241c13d00 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 10 client.16275 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 20 client.16275 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c8b410 0x563241c8d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:33.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 4 client.16275 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T23:47:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563241b7a800 con 0x563241c8b410 2024-04-24T23:47:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.934+0000 7f10e37fe640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c8b410 0x563241c8d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.935+0000 7f10c2ffd640 10 client.16275 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:33.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.935+0000 7f10e37fe640 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c8b410 0x563241c8d830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0c28c23c5f602f9 server_cookie=e026a01c8f1fa9a in_seq=0 out_seq=0 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10c2ffd640 1 -- 172.21.15.107:0/660038709 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563241b7a800 con 0x563241c8b410 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10c2ffd640 10 client.16275 handle_command_reply: tid=0 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 client.16275 shutdown 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 msgr2=0x563241c1ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc02f610 comp rx=0 tx=0).stop 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 shutdown_connections 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x563241c8b410 0x563241c8d830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563241c13d00 0x563241c1ebb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x563241c0eb90 0x563241c1dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10c2ffd640 0 client.16275 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 --2- 172.21.15.107:0/660038709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563241c0e1a0 0x563241c1d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 >> 172.21.15.107:0/660038709 conn(0x563241bd1950 msgr2=0x563241c040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 shutdown_connections 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 1 -- 172.21.15.107:0/660038709 wait complete. 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.936+0000 7f10ecb6f740 20 client.16275 trim_cache size 0 max 16384 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "e950dbb1-55dc-4faa-a681-5867cf42b62f", 2024-04-24T23:47:33.942 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-04-24T23:47:33.943 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:47:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 msgr2=0x7f10cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 0x7f10cc0a5050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f10d4005fd0 tx=0x7f10d4005850 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 msgr2=0x7f10e419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005e30 tx=0x7f10dc031ee0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 shutdown_connections 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f10cc0a2b90 0x7f10cc0a5050 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10e4103340 0x7f10e419f000 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f10e4102980 0x7f10e4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 --2- 172.21.15.107:0/1057191519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10e4101790 0x7f10e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 >> 172.21.15.107:0/1057191519 conn(0x7f10e40fcd20 msgr2=0x7f10e4105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 shutdown_connections 2024-04-24T23:47:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:33.937+0000 7f10e9e5d640 1 -- 172.21.15.107:0/1057191519 wait complete. 2024-04-24T23:47:33.948 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:47:33.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:47:34.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 -- 172.21.15.107:0/787069461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 msgr2=0x7fdd84100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 --2- 172.21.15.107:0/787069461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd84100fd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009900 tx=0x7fdd7402a410 comp rx=0 tx=0).stop 2024-04-24T23:47:34.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 -- 172.21.15.107:0/787069461 shutdown_connections 2024-04-24T23:47:34.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 --2- 172.21.15.107:0/787069461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd84108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 --2- 172.21.15.107:0/787069461 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 --2- 172.21.15.107:0/787069461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd84100fd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 -- 172.21.15.107:0/787069461 >> 172.21.15.107:0/787069461 conn(0x7fdd840fb380 msgr2=0x7fdd840fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 -- 172.21.15.107:0/787069461 shutdown_connections 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.048+0000 7fdd8a038640 1 -- 172.21.15.107:0/787069461 wait complete. 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 Processor -- start 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 -- start start 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd8419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8410c320 con 0x7fdd84100bb0 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8410c1a0 con 0x7fdd84102280 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd8a038640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8410c4a0 con 0x7fdd841018c0 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43696/0 (socket says 172.21.15.107:43696) 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 -- 172.21.15.107:0/2278013485 learned_addr learned my addr 172.21.15.107:0/2278013485 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd8419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 -- 172.21.15.107:0/2278013485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 msgr2=0x7fdd84196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 -- 172.21.15.107:0/2278013485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 msgr2=0x7fdd8419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd8419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd837fe640 1 -- 172.21.15.107:0/2278013485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8419e610 con 0x7fdd84100bb0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd8419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.049+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdd74031610 tx=0x7fdd740043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd80ff9640 1 -- 172.21.15.107:0/2278013485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7403f030 con 0x7fdd84100bb0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd80ff9640 1 -- 172.21.15.107:0/2278013485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd74003ce0 con 0x7fdd84100bb0 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8419e8a0 con 0x7fdd84100bb0 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd80ff9640 1 -- 172.21.15.107:0/2278013485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74035070 con 0x7fdd84100bb0 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.050+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8419ede0 con 0x7fdd84100bb0 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.052+0000 7fdd80ff9640 1 -- 172.21.15.107:0/2278013485 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdd74042030 con 0x7fdd84100bb0 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.052+0000 7fdd8ce29740 1 Processor -- start 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.052+0000 7fdd8ce29740 1 -- start start 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.052+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a92660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.052+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9a9a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5596f99f66b0 con 0x5596f9a92c30 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5596f999e880 con 0x5596f9a92240 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5596f99f2670 con 0x5596f9a97da0 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43706/0 (socket says 172.21.15.107:43706) 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 -- 172.21.15.107:0/2371901457 learned_addr learned my addr 172.21.15.107:0/2371901457 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd80ff9640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 0x7fdd6c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a92660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9a9a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 0x7fdd6c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 -- 172.21.15.107:0/2371901457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 msgr2=0x5596f9a9a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9a9a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 -- 172.21.15.107:0/2371901457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 msgr2=0x5596f9a92660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a92660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd80ff9640 1 -- 172.21.15.107:0/2278013485 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdd740efa30 con 0x7fdd84100bb0 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 -- 172.21.15.107:0/2371901457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f97d47d0 con 0x5596f9a92c30 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdd740bbbc0 tx=0x7fdd7402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e427d9dc5bd511 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 0x7fdd6c0a50d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009370 tx=0x7fdd7c009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd69ffb640 1 -- 172.21.15.107:0/2371901457 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74049020 con 0x5596f9a92c30 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd69ffb640 1 -- 172.21.15.107:0/2371901457 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd740bc5c0 con 0x5596f9a92c30 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.053+0000 7fdd69ffb640 1 -- 172.21.15.107:0/2371901457 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7403bd70 con 0x5596f9a92c30 2024-04-24T23:47:34.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2371901457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 msgr2=0x5596f9a97730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdd740bbbc0 tx=0x7fdd7402a410 comp rx=0 tx=0).stop 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2371901457 shutdown_connections 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9a9a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a97730 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2371901457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a92660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2371901457 >> 172.21.15.107:0/2371901457 conn(0x5596f9a8d8b0 msgr2=0x5596f9a8dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2371901457 shutdown_connections 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2371901457 wait complete. 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.054+0000 7fdd8ce29740 1 -- start start 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 Processor -- start 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a9c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9aa3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5596f99f66b0 con 0x5596f9a92c30 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5596f99b1f50 con 0x5596f9a92240 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5596f9a9ab10 con 0x5596f9a97da0 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd8ce29740 10 client.0 _ll_register_callbacks cb 0x5596f99ed000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43716/0 (socket says 172.21.15.107:43716) 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 -- 172.21.15.107:0/686540232 learned_addr learned my addr 172.21.15.107:0/686540232 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd83fff640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9aa3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd837fe640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a9c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 -- 172.21.15.107:0/686540232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 msgr2=0x5596f9aa3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9aa3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 -- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 msgr2=0x5596f9a9c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a9c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.055+0000 7fdd82ffd640 1 -- 172.21.15.107:0/686540232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f970a3f0 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdd78010670 tx=0x7fdd78010a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890c73e0fce7fa29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78012920 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdd78012ac0 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd740bc5c0 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd78025050 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 10 client.16287 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596f96ca0f0 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd740bc5c0 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdd7800f040 con 0x5596f9a92c30 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 10 client.16287 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 20 client.16287 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:34.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9aa8240 0x5596f9aaa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 4 client.16287 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.056+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596f9a03bf0 con 0x5596f9aa8240 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.057+0000 7fdd837fe640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9aa8240 0x5596f9aaa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.057+0000 7fdd837fe640 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9aa8240 0x5596f9aaa6a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a6e8c889604b95 server_cookie=1a4f3fee28a0667 in_seq=0 out_seq=0 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.057+0000 7fdd62ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd62ffd640 1 -- 172.21.15.107:0/686540232 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596f9a03bf0 con 0x5596f9aa8240 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd62ffd640 10 client.16287 handle_command_reply: tid=0 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 client.16287 shutdown 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 msgr2=0x5596f9a9c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdd78010670 tx=0x7fdd78010a70 comp rx=0 tx=0).stop 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 shutdown_connections 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9aa8240 0x5596f9aaa6a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a97da0 0x5596f9aa3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9a9c820 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd62ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/686540232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9a9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 >> 172.21.15.107:0/686540232 conn(0x5596f9a8d8b0 msgr2=0x5596f9a8caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 shutdown_connections 2024-04-24T23:47:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 1 -- 172.21.15.107:0/686540232 wait complete. 2024-04-24T23:47:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.058+0000 7fdd8ce29740 20 client.16287 trim_cache size 0 max 16384 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 Processor -- start 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 -- start start 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a92c30 0x5596f9aaef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a83a0 con 0x5596f9a8d8b0 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a9ee0 con 0x5596f9a92240 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a8a70 con 0x5596f9a92c30 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43724/0 (socket says 172.21.15.107:43724) 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 -- 172.21.15.107:0/1241989261 learned_addr learned my addr 172.21.15.107:0/1241989261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:34.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd83fff640 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a92c30 0x5596f9aaef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 -- 172.21.15.107:0/1241989261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a92c30 msgr2=0x5596f9aaef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a92c30 0x5596f9aaef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 -- 172.21.15.107:0/1241989261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 msgr2=0x5596f9aa3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 -- 172.21.15.107:0/1241989261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f9a8d3f0 con 0x5596f9a8d8b0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd837fe640 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdd740b8320 tx=0x7fdd7403b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35fcc55df46ebe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd697fa640 1 -- 172.21.15.107:0/1241989261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74049040 con 0x5596f9a8d8b0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd697fa640 1 -- 172.21.15.107:0/1241989261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd740bc5c0 con 0x5596f9a8d8b0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.076+0000 7fdd697fa640 1 -- 172.21.15.107:0/1241989261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd740b8530 con 0x5596f9a8d8b0 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- 172.21.15.107:0/1241989261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 msgr2=0x5596f9a8c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdd740b8320 tx=0x7fdd7403b870 comp rx=0 tx=0).stop 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- 172.21.15.107:0/1241989261 shutdown_connections 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a92c30 0x5596f9aaef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/1241989261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a8d8b0 0x5596f9a8c970 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- 172.21.15.107:0/1241989261 >> 172.21.15.107:0/1241989261 conn(0x5596f9a9bea0 msgr2=0x5596f9a8c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- 172.21.15.107:0/1241989261 shutdown_connections 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- 172.21.15.107:0/1241989261 wait complete. 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 -- start start 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.077+0000 7fdd8ce29740 1 Processor -- start 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a8d8b0 0x5596f9aa3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a9810 con 0x5596f9a92c30 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a9140 con 0x5596f9a92240 2024-04-24T23:47:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdd841a6380 con 0x5596f9a8d8b0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 10 client.0 _ll_register_callbacks cb 0x7fdd780094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43730/0 (socket says 172.21.15.107:43730) 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 -- 172.21.15.107:0/2452859489 learned_addr learned my addr 172.21.15.107:0/2452859489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a8d8b0 0x5596f9aa3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 -- 172.21.15.107:0/2452859489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a8d8b0 msgr2=0x5596f9aa3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a8d8b0 0x5596f9aa3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 -- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 msgr2=0x5596f9aa1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 -- 172.21.15.107:0/2452859489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f9a8e620 con 0x5596f9a92c30 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd82ffd640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd83fff640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c013380 tx=0x7fdd7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb8e8d58098ef61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:47:34.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c012d60 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdd7c00f480 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd740bc5c0 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd7c00f680 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.078+0000 7fdd8ce29740 10 client.16293 fetch_fsmap learned FSMap version 29 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596f9b0f570 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd740bc5c0 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdd7c02c930 con 0x5596f9a92c30 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 10 client.16293 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 20 client.16293 populate_metadata read hostname 'smithi107' 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9a97da0 0x5596f9b12010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 4 client.16293 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:47:34.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5596f99fdf40 con 0x5596f9a97da0 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.079+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9a97da0 0x5596f9b12010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd62ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd837fe640 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9a97da0 0x5596f9b12010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c281f846bbbb166 server_cookie=45493a30fff332d5 in_seq=0 out_seq=0 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd62ffd640 1 -- 172.21.15.107:0/2452859489 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5596f99fdf40 con 0x5596f9a97da0 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd62ffd640 10 client.16293 handle_command_reply: tid=0 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 client.16293 shutdown 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 msgr2=0x5596f9aa20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c013380 tx=0x7fdd7c004a40 comp rx=0 tx=0).stop 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 shutdown_connections 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x5596f9a97da0 0x5596f9b12010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596f9a92c30 0x5596f9aa20f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5596f9a92240 0x5596f9aa1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd62ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:47:34.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 --2- 172.21.15.107:0/2452859489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5596f9a8d8b0 0x5596f9aa3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 >> 172.21.15.107:0/2452859489 conn(0x5596f9a9bea0 msgr2=0x5596f9a910c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 shutdown_connections 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 1 -- 172.21.15.107:0/2452859489 wait complete. 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.080+0000 7fdd8ce29740 20 client.16293 trim_cache size 0 max 16384 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stdout: "e950dbb1-55dc-4faa-a681-5867cf42b62f": { 2024-04-24T23:47:34.087 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "e950dbb1-55dc-4faa-a681-5867cf42b62f", 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:47:34.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 msgr2=0x7fdd6c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 0x7fdd6c0a50d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009370 tx=0x7fdd7c009520 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 msgr2=0x7fdd841969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdd74031610 tx=0x7fdd740043b0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 shutdown_connections 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdd6c0a2c10 0x7fdd6c0a50d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd84102280 0x7fdd8419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdd841018c0 0x7fdd84196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 --2- 172.21.15.107:0/2278013485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd84100bb0 0x7fdd841969f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 >> 172.21.15.107:0/2278013485 conn(0x7fdd840fb380 msgr2=0x7fdd84104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 shutdown_connections 2024-04-24T23:47:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:47:34.081+0000 7fdd8a038640 1 -- 172.21.15.107:0/2278013485 wait complete. 2024-04-24T23:47:34.091 INFO:tasks.cephfs.filesystem:scrub status for tag:e950dbb1-55dc-4faa-a681-5867cf42b62f - {'path': '/', 'tag': 'e950dbb1-55dc-4faa-a681-5867cf42b62f', 'options': 'recursive,force'} 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: scrub queued for path: / 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: scrub summary: active paths [/] 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 705 MiB/s wr, 312 op/s 2024-04-24T23:47:35.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:35 smithi195 ceph-mon[33019]: scrub summary: idle 2024-04-24T23:47:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: scrub queued for path: / 2024-04-24T23:47:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: scrub summary: active paths [/] 2024-04-24T23:47:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 705 MiB/s wr, 312 op/s 2024-04-24T23:47:35.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:35 smithi170 ceph-mon[32714]: scrub summary: idle 2024-04-24T23:47:35.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: scrub queued for path: / 2024-04-24T23:47:35.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: scrub summary: active paths [/] 2024-04-24T23:47:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: scrub summary: idle+waiting paths [/] 2024-04-24T23:47:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 705 MiB/s wr, 312 op/s 2024-04-24T23:47:35.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:35 smithi107 ceph-mon[29118]: scrub summary: idle 2024-04-24T23:47:37.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:37 smithi195 ceph-mon[33019]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 831 MiB/s wr, 321 op/s 2024-04-24T23:47:37.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:37 smithi170 ceph-mon[32714]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 831 MiB/s wr, 321 op/s 2024-04-24T23:47:37.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:37 smithi107 ceph-mon[29118]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 831 MiB/s wr, 321 op/s 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 readers = 7905499.00 kB/sec 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 readers = 7903986.21 kB/sec 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 7905499.00 kB/sec 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 7905499.00 kB/sec 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 7905499.00 kB/sec 2024-04-24T23:47:38.545 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 10485760.00 kB 2024-04-24T23:47:39.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:39 smithi195 ceph-mon[33019]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 755 MiB/s wr, 300 op/s 2024-04-24T23:47:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:39 smithi170 ceph-mon[32714]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 755 MiB/s wr, 300 op/s 2024-04-24T23:47:39.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:39 smithi107 ceph-mon[29118]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 755 MiB/s wr, 300 op/s 2024-04-24T23:47:41.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:41 smithi195 ceph-mon[33019]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 639 MiB/s wr, 190 op/s 2024-04-24T23:47:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:41 smithi170 ceph-mon[32714]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 639 MiB/s wr, 190 op/s 2024-04-24T23:47:41.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:41 smithi107 ceph-mon[29118]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 639 MiB/s wr, 190 op/s 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: Children see throughput for 1 re-readers = 7899193.50 kB/sec 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: Parent sees throughput for 1 re-readers = 7897843.19 kB/sec 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: Min throughput per process = 7899193.50 kB/sec 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: Max throughput per process = 7899193.50 kB/sec 2024-04-24T23:47:41.954 INFO:tasks.workunit.client.0.smithi107.stdout: Avg throughput per process = 7899193.50 kB/sec 2024-04-24T23:47:41.955 INFO:tasks.workunit.client.0.smithi107.stdout: Min xfer = 10485760.00 kB 2024-04-24T23:47:43.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:43 smithi195 ceph-mon[33019]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 352 MiB/s wr, 99 op/s 2024-04-24T23:47:43.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:43 smithi170 ceph-mon[32714]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 352 MiB/s wr, 99 op/s 2024-04-24T23:47:43.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:43 smithi107 ceph-mon[29118]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 352 MiB/s wr, 99 op/s 2024-04-24T23:47:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:45 smithi195 ceph-mon[33019]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 285 MiB/s wr, 81 op/s 2024-04-24T23:47:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:45 smithi170 ceph-mon[32714]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 285 MiB/s wr, 81 op/s 2024-04-24T23:47:45.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:45 smithi107 ceph-mon[29118]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 285 MiB/s wr, 81 op/s 2024-04-24T23:47:47.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:47 smithi195 ceph-mon[33019]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 148 MiB/s wr, 50 op/s 2024-04-24T23:47:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:47 smithi170 ceph-mon[32714]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 148 MiB/s wr, 50 op/s 2024-04-24T23:47:47.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:47 smithi107 ceph-mon[29118]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 148 MiB/s wr, 50 op/s 2024-04-24T23:47:49.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:49 smithi195 ceph-mon[33019]: pgmap v443: 129 pgs: 129 active+clean; 9.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 24 op/s 2024-04-24T23:47:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:49 smithi170 ceph-mon[32714]: pgmap v443: 129 pgs: 129 active+clean; 9.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 24 op/s 2024-04-24T23:47:49.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:49 smithi107 ceph-mon[29118]: pgmap v443: 129 pgs: 129 active+clean; 9.4 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 24 op/s 2024-04-24T23:47:51.040 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:51.040 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:51.040 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-04-24T23:47:51.040 INFO:tasks.workunit.client.0.smithi107.stdout:iozone test complete. 2024-04-24T23:47:51.041 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T23:47:51.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-24T23:47:51.074 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-04-24T23:47:51.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T23:47:51.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:51 smithi107 ceph-mon[29118]: pgmap v444: 129 pgs: 129 active+clean; 2.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 166 op/s 2024-04-24T23:47:51.411 DEBUG:teuthology.parallel:result is None 2024-04-24T23:47:51.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T23:47:51.442 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T23:47:51.442 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-24T23:47:51.452 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-24T23:47:51.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:51 smithi195 ceph-mon[33019]: pgmap v444: 129 pgs: 129 active+clean; 2.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 166 op/s 2024-04-24T23:47:51.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:51 smithi170 ceph-mon[32714]: pgmap v444: 129 pgs: 129 active+clean; 2.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 166 op/s 2024-04-24T23:47:53.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:53 smithi195 ceph-mon[33019]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 193 op/s 2024-04-24T23:47:53.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:53 smithi170 ceph-mon[32714]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 193 op/s 2024-04-24T23:47:53.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:53 smithi107 ceph-mon[29118]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 KiB/s wr, 193 op/s 2024-04-24T23:47:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:55 smithi195 ceph-mon[33019]: pgmap v446: 129 pgs: 129 active+clean; 9.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 223 op/s 2024-04-24T23:47:55.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:55 smithi170 ceph-mon[32714]: pgmap v446: 129 pgs: 129 active+clean; 9.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 223 op/s 2024-04-24T23:47:55.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:55 smithi107 ceph-mon[29118]: pgmap v446: 129 pgs: 129 active+clean; 9.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 223 op/s 2024-04-24T23:47:57.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:57 smithi195 ceph-mon[33019]: pgmap v447: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-04-24T23:47:57.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:57 smithi170 ceph-mon[32714]: pgmap v447: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-04-24T23:47:57.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:57 smithi107 ceph-mon[29118]: pgmap v447: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-04-24T23:47:59.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:47:59 smithi195 ceph-mon[33019]: pgmap v448: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 224 op/s 2024-04-24T23:47:59.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:47:59 smithi170 ceph-mon[32714]: pgmap v448: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 224 op/s 2024-04-24T23:47:59.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:47:59 smithi107 ceph-mon[29118]: pgmap v448: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 224 op/s 2024-04-24T23:48:01.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:01 smithi195 ceph-mon[33019]: pgmap v449: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.7 KiB/s wr, 212 op/s 2024-04-24T23:48:01.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:01 smithi170 ceph-mon[32714]: pgmap v449: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.7 KiB/s wr, 212 op/s 2024-04-24T23:48:01.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:01 smithi107 ceph-mon[29118]: pgmap v449: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.7 KiB/s wr, 212 op/s 2024-04-24T23:48:03.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:03 smithi195 ceph-mon[33019]: pgmap v450: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 71 op/s 2024-04-24T23:48:03.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:03 smithi170 ceph-mon[32714]: pgmap v450: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 71 op/s 2024-04-24T23:48:03.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:03 smithi107 ceph-mon[29118]: pgmap v450: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 71 op/s 2024-04-24T23:48:04.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T23:48:04.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T23:48:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 -- 172.21.15.107:0/2181154067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170102250 msgr2=0x7f5170108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 --2- 172.21.15.107:0/2181154067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170102250 0x7f5170108ae0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5168009ca0 tx=0x7f516802a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 -- 172.21.15.107:0/2181154067 shutdown_connections 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 --2- 172.21.15.107:0/2181154067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170102250 0x7f5170108ae0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 --2- 172.21.15.107:0/2181154067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170101890 0x7f5170101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 --2- 172.21.15.107:0/2181154067 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f5170100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 -- 172.21.15.107:0/2181154067 >> 172.21.15.107:0/2181154067 conn(0x7f51700fbc10 msgr2=0x7f51700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 -- 172.21.15.107:0/2181154067 shutdown_connections 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.205+0000 7f5177a38640 1 -- 172.21.15.107:0/2181154067 wait complete. 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 Processor -- start 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 -- start start 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f517006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170102250 0x7f51701a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f517010c320 con 0x7f5170101890 2024-04-24T23:48:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f517010c1a0 con 0x7f5170102250 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5177a38640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f517010c4a0 con 0x7f51701006a0 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5174fd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5174fd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54626/0 (socket says 172.21.15.107:54626) 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5174fd3640 1 -- 172.21.15.107:0/3393825125 learned_addr learned my addr 172.21.15.107:0/3393825125 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.206+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170102250 0x7f51701a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 -- 172.21.15.107:0/3393825125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 msgr2=0x7f517006a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f51757d4640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f517006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f517006a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 -- 172.21.15.107:0/3393825125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170102250 msgr2=0x7f51701a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170102250 0x7f51701a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 -- 172.21.15.107:0/3393825125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51701a18f0 con 0x7f5170101890 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f51757d4640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f517006a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5174fd3640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f516000d950 tx=0x7f516000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f51667fc640 1 -- 172.21.15.107:0/3393825125 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160017070 con 0x7f5170101890 2024-04-24T23:48:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f51667fc640 1 -- 172.21.15.107:0/3393825125 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5160009c30 con 0x7f5170101890 2024-04-24T23:48:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51701a1b80 con 0x7f5170101890 2024-04-24T23:48:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f51667fc640 1 -- 172.21.15.107:0/3393825125 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516000f6b0 con 0x7f5170101890 2024-04-24T23:48:04.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.207+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51701a20d0 con 0x7f5170101890 2024-04-24T23:48:04.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.209+0000 7f51667fc640 1 -- 172.21.15.107:0/3393825125 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f51600109d0 con 0x7f5170101890 2024-04-24T23:48:04.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.209+0000 7f517a827740 1 Processor -- start 2024-04-24T23:48:04.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 -- start start 2024-04-24T23:48:04.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f1e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f23da0 0x55e940f26230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e940e826b0 con 0x55e940f1ec30 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e940e2a880 con 0x55e940f1e240 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f517a827740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e940e7e670 con 0x55e940f23da0 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51667fc640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 0x7f515c0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54634/0 (socket says 172.21.15.107:54634) 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 -- 172.21.15.107:0/4155158817 learned_addr learned my addr 172.21.15.107:0/4155158817 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 0x7f515c0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f5175fd5640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f1e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51667fc640 1 -- 172.21.15.107:0/3393825125 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f51600cf0e0 con 0x7f5170101890 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f5174fd3640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f23da0 0x55e940f26230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 -- 172.21.15.107:0/4155158817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f23da0 msgr2=0x55e940f26230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f23da0 0x55e940f26230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 -- 172.21.15.107:0/4155158817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 msgr2=0x55e940f1e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f1e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 -- 172.21.15.107:0/4155158817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e940c607d0 con 0x55e940f1ec30 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f5175fd5640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f1e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.210+0000 7f51757d4640 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f516c009c30 tx=0x7f516c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8c1e21d1ef919f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f515b7fe640 1 -- 172.21.15.107:0/4155158817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c016070 con 0x55e940f1ec30 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 0x7f515c0a5080 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5168004af0 tx=0x7f51680095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f515b7fe640 1 -- 172.21.15.107:0/4155158817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f516c005500 con 0x55e940f1ec30 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f515b7fe640 1 -- 172.21.15.107:0/4155158817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c0116d0 con 0x55e940f1ec30 2024-04-24T23:48:04.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 -- 172.21.15.107:0/4155158817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 msgr2=0x55e940f23730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f516c009c30 tx=0x7f516c00e6c0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 -- 172.21.15.107:0/4155158817 shutdown_connections 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f23da0 0x55e940f26230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1ec30 0x55e940f23730 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 --2- 172.21.15.107:0/4155158817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f1e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 -- 172.21.15.107:0/4155158817 >> 172.21.15.107:0/4155158817 conn(0x55e940f198b0 msgr2=0x55e940f19cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 -- 172.21.15.107:0/4155158817 shutdown_connections 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.211+0000 7f517a827740 1 -- 172.21.15.107:0/4155158817 wait complete. 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 -- start start 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 Processor -- start 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f28b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f23da0 0x55e940f2fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e940f26d50 con 0x55e940f1e240 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e940e3df50 con 0x55e940f23da0 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55e940f26ed0 con 0x55e940f1ec30 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f517a827740 10 client.0 _ll_register_callbacks cb 0x55e940e79000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f51757d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f51757d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54642/0 (socket says 172.21.15.107:54642) 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f51757d4640 1 -- 172.21.15.107:0/2913573273 learned_addr learned my addr 172.21.15.107:0/2913573273 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f5175fd5640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f23da0 0x55e940f2fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.212+0000 7f51757d4640 1 -- 172.21.15.107:0/2913573273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 msgr2=0x55e940f28b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f51757d4640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f28b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f51757d4640 1 -- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f23da0 msgr2=0x55e940f2fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f51757d4640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f23da0 0x55e940f2fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f51757d4640 1 -- 172.21.15.107:0/2913573273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e940b963f0 con 0x55e940f1e240 2024-04-24T23:48:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f51757d4640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f516c005e70 tx=0x7f516c00efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ba701185400fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c028060 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f516c005020 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f516c005500 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f516c02f050 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 10 client.16305 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e940b560f0 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f516c005500 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f516c00f040 con 0x55e940f1e240 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 10 client.16305 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 20 client.16305 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f34140 0x55e940f365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 4 client.16305 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.213+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e940e8fbf0 con 0x55e940f34140 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.214+0000 7f5174fd3640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f34140 0x55e940f365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.214+0000 7f5158ff9640 10 client.16305 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.214+0000 7f5174fd3640 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f34140 0x55e940f365d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b90ed880172956c server_cookie=dcfbf11bcd387942 in_seq=0 out_seq=0 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f5158ff9640 1 -- 172.21.15.107:0/2913573273 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e940e8fbf0 con 0x55e940f34140 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f5158ff9640 10 client.16305 handle_command_reply: tid=0 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 client.16305 shutdown 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 msgr2=0x55e940f28640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f516c005e70 tx=0x7f516c00efc0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 shutdown_connections 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f34140 0x55e940f365d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f23da0 0x55e940f2fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f28b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f5158ff9640 0 client.16305 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 --2- 172.21.15.107:0/2913573273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f1e240 0x55e940f28640 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 >> 172.21.15.107:0/2913573273 conn(0x55e940f198b0 msgr2=0x55e940f18af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 shutdown_connections 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 1 -- 172.21.15.107:0/2913573273 wait complete. 2024-04-24T23:48:04.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.215+0000 7f517a827740 20 client.16305 trim_cache size 0 max 16384 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 Processor -- start 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 -- start start 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f18260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51701ac700 con 0x55e940f23da0 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f51701ae240 con 0x55e940f1e240 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f517a827740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f51701acdd0 con 0x55e940f1ec30 2024-04-24T23:48:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54656/0 (socket says 172.21.15.107:54656) 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 -- 172.21.15.107:0/3570477965 learned_addr learned my addr 172.21.15.107:0/3570477965 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5174fd3640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f51757d4640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f18260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 -- 172.21.15.107:0/3570477965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 msgr2=0x55e940f2fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 -- 172.21.15.107:0/3570477965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 msgr2=0x55e940f18260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f18260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 -- 172.21.15.107:0/3570477965 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e940f0b090 con 0x55e940f23da0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5174fd3640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.233+0000 7f5175fd5640 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5168035000 tx=0x7f516803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bbfc42da6e40d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f515affd640 1 -- 172.21.15.107:0/3570477965 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516802fb90 con 0x55e940f23da0 2024-04-24T23:48:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f515affd640 1 -- 172.21.15.107:0/3570477965 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5168036040 con 0x55e940f23da0 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f515affd640 1 -- 172.21.15.107:0/3570477965 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516803b780 con 0x55e940f23da0 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- 172.21.15.107:0/3570477965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 msgr2=0x55e940f187a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5168035000 tx=0x7f516803d000 comp rx=0 tx=0).stop 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- 172.21.15.107:0/3570477965 shutdown_connections 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f187a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 --2- 172.21.15.107:0/3570477965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f18260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- 172.21.15.107:0/3570477965 >> 172.21.15.107:0/3570477965 conn(0x55e940f198b0 msgr2=0x55e940f1fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- 172.21.15.107:0/3570477965 shutdown_connections 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- 172.21.15.107:0/3570477965 wait complete. 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.234+0000 7f517a827740 1 -- start start 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 Processor -- start 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f2cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51701adb70 con 0x55e940f23da0 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f51701ad4a0 con 0x55e940f1e240 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f51701aa6e0 con 0x55e940f1ec30 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f517a827740 10 client.0 _ll_register_callbacks cb 0x7f516c009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f51757d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f2cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f51757d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f2cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.107:41594/0 (socket says 172.21.15.107:41594) 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f51757d4640 1 -- 172.21.15.107:0/1862460179 learned_addr learned my addr 172.21.15.107:0/1862460179 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54668/0 (socket says 172.21.15.107:54668) 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5174fd3640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 -- 172.21.15.107:0/1862460179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 msgr2=0x55e940f2d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 -- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 msgr2=0x55e940f2cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f2cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.235+0000 7f5175fd5640 1 -- 172.21.15.107:0/1862460179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e940f2e790 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5175fd5640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f516802ade0 tx=0x7f5168030d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68f45adb61548d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516802fda0 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f51680305c0 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5168036040 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51680307c0 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 10 client.16311 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e940f2fa10 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5168036040 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5168047070 con 0x55e940f23da0 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 10 client.16311 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:48:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 20 client.16311 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f9aa80 0x55e940f9ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 4 client.16311 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.236+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e940e89f40 con 0x55e940f9aa80 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f51757d4640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f9aa80 0x55e940f9ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f51757d4640 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f9aa80 0x55e940f9ce70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3ac7a967299dc9e server_cookie=bd22a069641b2a31 in_seq=0 out_seq=0 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f5158ff9640 10 client.16311 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f5158ff9640 1 -- 172.21.15.107:0/1862460179 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e940e89f40 con 0x55e940f9aa80 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f5158ff9640 10 client.16311 handle_command_reply: tid=0 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 client.16311 shutdown 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 msgr2=0x55e940f2e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f516802ade0 tx=0x7f5168030d30 comp rx=0 tx=0).stop 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 shutdown_connections 2024-04-24T23:48:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55e940f9aa80 0x55e940f9ce70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e940f23da0 0x55e940f2e1a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f5158ff9640 0 client.16311 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55e940f1ec30 0x55e940f2d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 --2- 172.21.15.107:0/1862460179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e940f1e240 0x55e940f2cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 >> 172.21.15.107:0/1862460179 conn(0x55e940f198b0 msgr2=0x55e940f1c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 shutdown_connections 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 1 -- 172.21.15.107:0/1862460179 wait complete. 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.237+0000 7f517a827740 20 client.16311 trim_cache size 0 max 16384 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-04-24T23:48:04.244 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-04-24T23:48:04.245 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 msgr2=0x7f515c0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 0x7f515c0a5080 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5168004af0 tx=0x7f51680095c0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 msgr2=0x7f517019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f516000d950 tx=0x7f516000de10 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 shutdown_connections 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f515c0a2bc0 0x7f515c0a5080 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5170102250 0x7f51701a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5170101890 0x7f517019ebb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 --2- 172.21.15.107:0/3393825125 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f51701006a0 0x7f517006a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 >> 172.21.15.107:0/3393825125 conn(0x7f51700fbc10 msgr2=0x7f5170104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 shutdown_connections 2024-04-24T23:48:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.238+0000 7f5177a38640 1 -- 172.21.15.107:0/3393825125 wait complete. 2024-04-24T23:48:04.249 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T23:48:04.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:48:04.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- 172.21.15.107:0/542680312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2501033f0 msgr2=0x7fd250109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 --2- 172.21.15.107:0/542680312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2501033f0 0x7fd250109c80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd248009cd0 tx=0x7fd24802a8e0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- 172.21.15.107:0/542680312 shutdown_connections 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 --2- 172.21.15.107:0/542680312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2501033f0 0x7fd250109c80 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 --2- 172.21.15.107:0/542680312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd250102a30 0x7fd250102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 --2- 172.21.15.107:0/542680312 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2500ffa60 0x7fd2500ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- 172.21.15.107:0/542680312 >> 172.21.15.107:0/542680312 conn(0x7fd2500fb3b0 msgr2=0x7fd2500fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- 172.21.15.107:0/542680312 shutdown_connections 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- 172.21.15.107:0/542680312 wait complete. 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 Processor -- start 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.348+0000 7fd256948640 1 -- start start 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 0x7fd250197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 0x7fd25019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010d460 con 0x7fd250102a30 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010d2e0 con 0x7fd2500ffa60 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd256948640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010d5e0 con 0x7fd2501033f0 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54676/0 (socket says 172.21.15.107:54676) 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 -- 172.21.15.107:0/2667294597 learned_addr learned my addr 172.21.15.107:0/2667294597 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24ffff640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 0x7fd250197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd254ee5640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 0x7fd25019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 -- 172.21.15.107:0/2667294597 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 msgr2=0x7fd25019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 0x7fd25019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 -- 172.21.15.107:0/2667294597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 msgr2=0x7fd250197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 0x7fd250197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 -- 172.21.15.107:0/2667294597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25019f690 con 0x7fd250102a30 2024-04-24T23:48:04.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd254ee5640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 0x7fd25019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24ffff640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 0x7fd250197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24f7fe640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c890 tx=0x7fd24400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.349+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244004220 con 0x7fd250102a30 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.350+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2440043c0 con 0x7fd250102a30 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.350+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25019f980 con 0x7fd250102a30 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.350+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244004a00 con 0x7fd250102a30 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.350+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25019fed0 con 0x7fd250102a30 2024-04-24T23:48:04.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.350+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2500689b0 con 0x7fd250102a30 2024-04-24T23:48:04.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.352+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd2440079b0 con 0x7fd250102a30 2024-04-24T23:48:04.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.352+0000 7fd24d7fa640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 0x7fd2380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.352+0000 7fd24ffff640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 0x7fd2380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.352+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd2440cf5c0 con 0x7fd250102a30 2024-04-24T23:48:04.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.353+0000 7fd24ffff640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 0x7fd2380a5050 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd2400058c0 tx=0x7fd240005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.356+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd244014030 con 0x7fd250102a30 2024-04-24T23:48:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.567+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd25002d170 con 0x7fd250102a30 2024-04-24T23:48:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.568+0000 7fd24d7fa640 1 -- 172.21.15.107:0/2667294597 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7fd2440ce090 con 0x7fd250102a30 2024-04-24T23:48:04.568 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:04.568 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:48:04.569 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:48:04.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 msgr2=0x7fd2380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 0x7fd2380a5050 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd2400058c0 tx=0x7fd240005850 comp rx=0 tx=0).stop 2024-04-24T23:48:04.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 msgr2=0x7fd250197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c890 tx=0x7fd24400cd50 comp rx=0 tx=0).stop 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 shutdown_connections 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd2380a2b90 0x7fd2380a5050 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd2501033f0 0x7fd25019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd250102a30 0x7fd250197fb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 --2- 172.21.15.107:0/2667294597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2500ffa60 0x7fd250197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 >> 172.21.15.107:0/2667294597 conn(0x7fd2500fb3b0 msgr2=0x7fd250105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 shutdown_connections 2024-04-24T23:48:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.570+0000 7fd256948640 1 -- 172.21.15.107:0/2667294597 wait complete. 2024-04-24T23:48:04.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/3811903627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54101870 msgr2=0x7f5e54101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/3811903627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54101870 0x7f5e54101cf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009900 tx=0x7f5e4002a410 comp rx=0 tx=0).stop 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/3811903627 shutdown_connections 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/3811903627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e54102230 0x7f5e54108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/3811903627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54101870 0x7f5e54101cf0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/3811903627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e54100680 0x7f5e54100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/3811903627 >> 172.21.15.107:0/3811903627 conn(0x7f5e540fbbf0 msgr2=0x7f5e540fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/3811903627 shutdown_connections 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.684+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/3811903627 wait complete. 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 Processor -- start 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 -- start start 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e54101870 0x7f5e5419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e54102230 0x7f5e5419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5410c3c0 con 0x7f5e54100680 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5410c240 con 0x7f5e54101870 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e5ac2a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5410c540 con 0x7f5e54102230 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54694/0 (socket says 172.21.15.107:54694) 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 -- 172.21.15.107:0/1502589024 learned_addr learned my addr 172.21.15.107:0/1502589024 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 -- 172.21.15.107:0/1502589024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e54102230 msgr2=0x7f5e5419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e54102230 0x7f5e5419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 -- 172.21.15.107:0/1502589024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e54101870 msgr2=0x7f5e5419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e54101870 0x7f5e5419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 -- 172.21.15.107:0/1502589024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5419f790 con 0x7f5e54100680 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e589c6640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800b450 tx=0x7f5e4800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.685+0000 7f5e51ffb640 1 -- 172.21.15.107:0/1502589024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48015020 con 0x7f5e54100680 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.686+0000 7f5e51ffb640 1 -- 172.21.15.107:0/1502589024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e48011d70 con 0x7f5e54100680 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.686+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5419fa20 con 0x7f5e54100680 2024-04-24T23:48:04.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.686+0000 7f5e51ffb640 1 -- 172.21.15.107:0/1502589024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4800f340 con 0x7f5e54100680 2024-04-24T23:48:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.686+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e5419ffa0 con 0x7f5e54100680 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e51ffb640 1 -- 172.21.15.107:0/1502589024 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5e48018020 con 0x7f5e54100680 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 Processor -- start 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 -- start start 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e51ffb640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 0x7f5e440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef6aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef72630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5565feeca670 con 0x5565fef6b030 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5565fee72970 con 0x5565fef701a0 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e5da15740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5565feec6630 con 0x5565fef6a640 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e51ffb640 1 -- 172.21.15.107:0/1502589024 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5e480ceff0 con 0x7f5e54100680 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 0x7f5e440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54710/0 (socket says 172.21.15.107:54710) 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.688+0000 7f5e53fff640 1 -- 172.21.15.107:0/1257528579 learned_addr learned my addr 172.21.15.107:0/1257528579 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef72630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e589c6640 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef6aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 -- 172.21.15.107:0/1257528579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 msgr2=0x5565fef6aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef6aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 -- 172.21.15.107:0/1257528579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 msgr2=0x5565fef72630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef72630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 -- 172.21.15.107:0/1257528579 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565fee96400 con 0x5565fef6b030 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 0x7f5e440a5050 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005fd0 tx=0x7f5e4c0138e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e53fff640 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002a930 tx=0x7f5e40000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec63b2bc594be49a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e3effd640 1 -- 172.21.15.107:0/1257528579 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4003e030 con 0x5565fef6b030 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e3effd640 1 -- 172.21.15.107:0/1257528579 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e400042e0 con 0x5565fef6b030 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.689+0000 7f5e3effd640 1 -- 172.21.15.107:0/1257528579 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40035040 con 0x5565fef6b030 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- 172.21.15.107:0/1257528579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 msgr2=0x5565fef6fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002a930 tx=0x7f5e40000bc0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- 172.21.15.107:0/1257528579 shutdown_connections 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef72630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6b030 0x5565fef6fb30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1257528579 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef6aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- 172.21.15.107:0/1257528579 >> 172.21.15.107:0/1257528579 conn(0x5565fef618a0 msgr2=0x5565fef61ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- 172.21.15.107:0/1257528579 shutdown_connections 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- 172.21.15.107:0/1257528579 wait complete. 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 -- start start 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.690+0000 7f5e5da15740 1 Processor -- start 2024-04-24T23:48:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 0x5565fef74bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef7bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5565fef730f0 con 0x5565fef6a640 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5565fee86050 con 0x5565fef701a0 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5565fef73270 con 0x5565fef6b030 2024-04-24T23:48:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 10 client.0 _ll_register_callbacks cb 0x5565febddbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54720/0 (socket says 172.21.15.107:54720) 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 -- 172.21.15.107:0/2752833748 learned_addr learned my addr 172.21.15.107:0/2752833748 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e53fff640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 0x5565fef74bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e591c7640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef7bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 -- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 msgr2=0x5565fef74bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 0x5565fef74bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 -- 172.21.15.107:0/2752833748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 msgr2=0x5565fef7bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef7bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 -- 172.21.15.107:0/2752833748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565fee57dc0 con 0x5565fef6a640 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e53fff640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 0x5565fef74bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5e4809b140 tx=0x7f5e48009340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c85589d7ae59b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e33fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4809f7f0 con 0x5565fef6a640 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5e48010570 con 0x5565fef6a640 2024-04-24T23:48:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e400042e0 con 0x5565fef6a640 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e480a40b0 con 0x5565fef6a640 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 10 client.16329 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565febde3f0 con 0x5565fef6a640 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.691+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e400042e0 con 0x5565fef6a640 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5e480b41f0 con 0x5565fef6a640 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 10 client.16329 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 10 client.16329 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 20 client.16329 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fef80540 0x5565fef829b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 4 client.16329 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565feed7bb0 con 0x5565fef80540 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e53fff640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fef80540 0x5565fef829b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e33fff640 10 client.16329 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.692+0000 7f5e53fff640 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fef80540 0x5565fef829b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=57e8da4291007f7e server_cookie=cd5429556b7e1652 in_seq=0 out_seq=0 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e33fff640 1 -- 172.21.15.107:0/2752833748 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565feed7bb0 con 0x5565fef80540 2024-04-24T23:48:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e33fff640 10 client.16329 handle_command_reply: tid=0 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 client.16329 shutdown 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 msgr2=0x5565fef746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5e4809b140 tx=0x7f5e48009340 comp rx=0 tx=0).stop 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 shutdown_connections 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fef80540 0x5565fef829b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef701a0 0x5565fef7bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e33fff640 0 client.16329 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6b030 0x5565fef74bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2752833748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef6a640 0x5565fef746b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 >> 172.21.15.107:0/2752833748 conn(0x5565fef618a0 msgr2=0x5565fef60b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 shutdown_connections 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.693+0000 7f5e5da15740 1 -- 172.21.15.107:0/2752833748 wait complete. 2024-04-24T23:48:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.694+0000 7f5e5da15740 20 client.16329 trim_cache size 0 max 16384 2024-04-24T23:48:04.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.706+0000 7f5e5da15740 1 Processor -- start 2024-04-24T23:48:04.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.706+0000 7f5e5da15740 1 -- start start 2024-04-24T23:48:04.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef647b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a81d0 con 0x5565fef701a0 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a9d10 con 0x5565fef6b030 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e5da15740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a88a0 con 0x5565fef6a640 2024-04-24T23:48:04.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54736/0 (socket says 172.21.15.107:54736) 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 -- 172.21.15.107:0/1279235097 learned_addr learned my addr 172.21.15.107:0/1279235097 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e589c6640 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e53fff640 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 -- 172.21.15.107:0/1279235097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 msgr2=0x5565fef647b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef647b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 -- 172.21.15.107:0/1279235097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 msgr2=0x5565fef7bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 -- 172.21.15.107:0/1279235097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565feca87d0 con 0x5565fef701a0 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e591c7640 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0105f0 tx=0x7f5e4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62793e9c02179a59 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e3e7fc640 1 -- 172.21.15.107:0/1279235097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c004d20 con 0x5565fef701a0 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e3e7fc640 1 -- 172.21.15.107:0/1279235097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4c011c10 con 0x5565fef701a0 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.707+0000 7f5e3e7fc640 1 -- 172.21.15.107:0/1279235097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c005960 con 0x5565fef701a0 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- 172.21.15.107:0/1279235097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 msgr2=0x5565fef64cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0105f0 tx=0x7f5e4c016000 comp rx=0 tx=0).stop 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- 172.21.15.107:0/1279235097 shutdown_connections 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef64cf0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 --2- 172.21.15.107:0/1279235097 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef647b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- 172.21.15.107:0/1279235097 >> 172.21.15.107:0/1279235097 conn(0x5565fef618a0 msgr2=0x5565fef60b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- 172.21.15.107:0/1279235097 shutdown_connections 2024-04-24T23:48:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- 172.21.15.107:0/1279235097 wait complete. 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 -- start start 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.708+0000 7f5e5da15740 1 Processor -- start 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef79b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a9640 con 0x5565fef701a0 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a8f70 con 0x5565fef6b030 2024-04-24T23:48:04.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541a61b0 con 0x5565fef6a640 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 10 client.0 _ll_register_callbacks cb 0x7f5e480976e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54740/0 (socket says 172.21.15.107:54740) 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 -- 172.21.15.107:0/2406618222 learned_addr learned my addr 172.21.15.107:0/2406618222 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef79b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e53fff640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 -- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 msgr2=0x5565fef79b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef79b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 -- 172.21.15.107:0/2406618222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 msgr2=0x5565fef7a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 -- 172.21.15.107:0/2406618222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565fef7b590 con 0x5565fef701a0 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e591c7640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005ed0 tx=0x7f5e4c0056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d21de6e809d9972 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e33fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c037060 con 0x5565fef701a0 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5e4c011730 con 0x5565fef701a0 2024-04-24T23:48:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4c011c10 con 0x5565fef701a0 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e4c03e050 con 0x5565fef701a0 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 10 client.16335 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.709+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565fefe7160 con 0x5565fef701a0 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4c011c10 con 0x5565fef701a0 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5e4c00f040 con 0x5565fef701a0 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 10 client.16335 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 10 client.16335 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 20 client.16335 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fefe7b30 0x5565fefe9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 4 client.16335 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 --> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565feed1f00 con 0x5565fefe7b30 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fefe7b30 0x5565fefe9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e33fff640 10 client.16335 ms_handle_connect on v2:172.21.15.195:6834/1997811297 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.710+0000 7f5e589c6640 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fefe7b30 0x5565fefe9f50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6fa42f5f49b54ade server_cookie=f2f71e4caff3b2f8 in_seq=0 out_seq=0 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e33fff640 1 -- 172.21.15.107:0/2406618222 <== mds.4 v2:172.21.15.195:6834/1997811297 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565feed1f00 con 0x5565fefe7b30 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e33fff640 10 client.16335 handle_command_reply: tid=0 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 client.16335 shutdown 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 msgr2=0x5565fef7afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005ed0 tx=0x7f5e4c0056e0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 shutdown_connections 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:6834/1997811297,v1:172.21.15.195:6835/1997811297] conn(0x5565fefe7b30 0x5565fefe9f50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5565fef701a0 0x5565fef7afa0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5565fef6b030 0x5565fef7a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e33fff640 0 client.16335 ms_handle_reset on v2:172.21.15.195:6834/1997811297 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 --2- 172.21.15.107:0/2406618222 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5565fef6a640 0x5565fef79b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 >> 172.21.15.107:0/2406618222 conn(0x5565fef618a0 msgr2=0x5565fef602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 shutdown_connections 2024-04-24T23:48:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 1 -- 172.21.15.107:0/2406618222 wait complete. 2024-04-24T23:48:04.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.711+0000 7f5e5da15740 20 client.16335 trim_cache size 0 max 16384 2024-04-24T23:48:04.736 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:48:04.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 msgr2=0x7f5e440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 0x7f5e440a5050 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005fd0 tx=0x7f5e4c0138e0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 msgr2=0x7f5e54069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800b450 tx=0x7f5e4800b910 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 shutdown_connections 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f5e440a2b90 0x7f5e440a5050 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5e54102230 0x7f5e5419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e54101870 0x7f5e5419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 --2- 172.21.15.107:0/1502589024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e54100680 0x7f5e54069440 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 >> 172.21.15.107:0/1502589024 conn(0x7f5e540fbbf0 msgr2=0x7f5e54104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 shutdown_connections 2024-04-24T23:48:04.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.712+0000 7f5e5ac2a640 1 -- 172.21.15.107:0/1502589024 wait complete. 2024-04-24T23:48:04.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-24T23:48:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 -- 172.21.15.107:0/1196490626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 msgr2=0x7f5314109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 --2- 172.21.15.107:0/1196490626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314109bd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f530c009ca0 tx=0x7f530c02a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 -- 172.21.15.107:0/1196490626 shutdown_connections 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 --2- 172.21.15.107:0/1196490626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314109bd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 --2- 172.21.15.107:0/1196490626 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314102980 0x7f5314102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 --2- 172.21.15.107:0/1196490626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314101790 0x7f5314101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 -- 172.21.15.107:0/1196490626 >> 172.21.15.107:0/1196490626 conn(0x7f53140fcd20 msgr2=0x7f53140ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 -- 172.21.15.107:0/1196490626 shutdown_connections 2024-04-24T23:48:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.840+0000 7f531aed2640 1 -- 172.21.15.107:0/1196490626 wait complete. 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 Processor -- start 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 -- start start 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314101790 0x7f53140754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314102980 0x7f5314073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f531410d410 con 0x7f5314103340 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f531410d290 con 0x7f5314102980 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531aed2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f531410d590 con 0x7f5314101790 2024-04-24T23:48:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54758/0 (socket says 172.21.15.107:54758) 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 -- 172.21.15.107:0/3544225172 learned_addr learned my addr 172.21.15.107:0/3544225172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f5318c6e640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314101790 0x7f53140754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f5313fff640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314102980 0x7f5314073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 -- 172.21.15.107:0/3544225172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314101790 msgr2=0x7f53140754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314101790 0x7f53140754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 -- 172.21.15.107:0/3544225172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314102980 msgr2=0x7f5314073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314102980 0x7f5314073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 -- 172.21.15.107:0/3544225172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314074900 con 0x7f5314103340 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.841+0000 7f531946f640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f530c031ae0 tx=0x7f530c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.842+0000 7f5311ffb640 1 -- 172.21.15.107:0/3544225172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c003c30 con 0x7f5314103340 2024-04-24T23:48:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.842+0000 7f5311ffb640 1 -- 172.21.15.107:0/3544225172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530c003dd0 con 0x7f5314103340 2024-04-24T23:48:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.842+0000 7f5311ffb640 1 -- 172.21.15.107:0/3544225172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c02fcb0 con 0x7f5314103340 2024-04-24T23:48:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.842+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53141a1ba0 con 0x7f5314103340 2024-04-24T23:48:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.842+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53141a2140 con 0x7f5314103340 2024-04-24T23:48:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f5311ffb640 1 -- 172.21.15.107:0/3544225172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f530c004050 con 0x7f5314103340 2024-04-24T23:48:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 Processor -- start 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 -- start start 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5562046a2d90 0x5562046a5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556204601670 con 0x55620469d230 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5562045a9970 con 0x5562046a2d90 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f531dbde740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5562045fd630 con 0x55620469dc20 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f5311ffb640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 0x7f53040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54766/0 (socket says 172.21.15.107:54766) 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.844+0000 7f5313fff640 1 -- 172.21.15.107:0/2900460605 learned_addr learned my addr 172.21.15.107:0/2900460605 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f531946f640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 0x7f53040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5562046a2d90 0x5562046a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5311ffb640 1 -- 172.21.15.107:0/3544225172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f530c0f0660 con 0x7f5314103340 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5318c6e640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 -- 172.21.15.107:0/2900460605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 msgr2=0x5562046a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 -- 172.21.15.107:0/2900460605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5562046a2d90 msgr2=0x5562046a5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5562046a2d90 0x5562046a5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 -- 172.21.15.107:0/2900460605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562045cd400 con 0x55620469d230 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5318c6e640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f5313fff640 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f53000124f0 tx=0x7f530000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4f1c195642fec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f531946f640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 0x7f53040a50d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f530c0edde0 tx=0x7f530c0b20f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f52feffd640 1 -- 172.21.15.107:0/2900460605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300018030 con 0x55620469d230 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f52feffd640 1 -- 172.21.15.107:0/2900460605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5300012110 con 0x55620469d230 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.845+0000 7f52feffd640 1 -- 172.21.15.107:0/2900460605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300004e60 con 0x55620469d230 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- 172.21.15.107:0/2900460605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 msgr2=0x55620469d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f53000124f0 tx=0x7f530000c830 comp rx=0 tx=0).stop 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- 172.21.15.107:0/2900460605 shutdown_connections 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5562046a2d90 0x5562046a5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 --2- 172.21.15.107:0/2900460605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469d230 0x55620469d650 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- 172.21.15.107:0/2900460605 >> 172.21.15.107:0/2900460605 conn(0x5562046988a0 msgr2=0x556204698ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- 172.21.15.107:0/2900460605 shutdown_connections 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- 172.21.15.107:0/2900460605 wait complete. 2024-04-24T23:48:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 -- start start 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.846+0000 7f531dbde740 1 Processor -- start 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5562046a5d40 con 0x5562046a2d90 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5562045bd050 con 0x55620469d230 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5562046a5ec0 con 0x55620469dc20 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 10 client.0 _ll_register_callbacks cb 0x556204314be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54778/0 (socket says 172.21.15.107:54778) 2024-04-24T23:48:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 -- 172.21.15.107:0/3100569886 learned_addr learned my addr 172.21.15.107:0/3100569886 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5318c6e640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 -- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 msgr2=0x5562046a78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 -- 172.21.15.107:0/3100569886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 msgr2=0x5562046a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 -- 172.21.15.107:0/3100569886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55620458edc0 con 0x5562046a2d90 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5318c6e640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f5313fff640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5300004fc0 tx=0x7f5300012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4913e91050b027 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f52f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300018040 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f53000054d0 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5300012110 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f530000e620 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 10 client.16347 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.847+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5562043153f0 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5300012110 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5300029060 con 0x5562046a2d90 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 10 client.16347 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 10 client.16347 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 20 client.16347 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562046b2e50 0x5562046b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 4 client.16347 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55620460ebb0 con 0x5562046b2e50 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f5318c6e640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562046b2e50 0x5562046b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f52f3fff640 10 client.16347 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.848+0000 7f5318c6e640 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562046b2e50 0x5562046b5280 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fde0281d17568794 server_cookie=4633fd25e592abf2 in_seq=0 out_seq=0 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f52f3fff640 1 -- 172.21.15.107:0/3100569886 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55620460ebb0 con 0x5562046b2e50 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f52f3fff640 10 client.16347 handle_command_reply: tid=0 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 client.16347 shutdown 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 msgr2=0x5562046ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5300004fc0 tx=0x7f5300012f70 comp rx=0 tx=0).stop 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 shutdown_connections 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562046b2e50 0x5562046b5280 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ae930 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f52f3fff640 0 client.16347 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 --2- 172.21.15.107:0/3100569886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 >> 172.21.15.107:0/3100569886 conn(0x5562046988a0 msgr2=0x556204697b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 shutdown_connections 2024-04-24T23:48:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 1 -- 172.21.15.107:0/3100569886 wait complete. 2024-04-24T23:48:04.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.849+0000 7f531dbde740 20 client.16347 trim_cache size 0 max 16384 2024-04-24T23:48:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.862+0000 7f531dbde740 1 Processor -- start 2024-04-24T23:48:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.862+0000 7f531dbde740 1 -- start start 2024-04-24T23:48:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.862+0000 7f531dbde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x55620469bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f531dbde740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f531dbde740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5562046a2d90 0x5562046ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f531dbde740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53141ac750 con 0x55620469dc20 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f531dbde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f53141ae290 con 0x55620469d230 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f531dbde740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f53141ace20 con 0x5562046a2d90 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54792/0 (socket says 172.21.15.107:54792) 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 -- 172.21.15.107:0/1309025056 learned_addr learned my addr 172.21.15.107:0/1309025056 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5313fff640 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x55620469bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5313fff640 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5562046a2d90 0x5562046ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 -- 172.21.15.107:0/1309025056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5562046a2d90 msgr2=0x5562046ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5562046a2d90 0x5562046ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 -- 172.21.15.107:0/1309025056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 msgr2=0x55620469bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x55620469bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 -- 172.21.15.107:0/1309025056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562043df7d0 con 0x55620469dc20 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f5318c6e640 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5308009d40 tx=0x7f5308010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b5ca890d3474c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f52fe7fc640 1 -- 172.21.15.107:0/1309025056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308021020 con 0x55620469dc20 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f52fe7fc640 1 -- 172.21.15.107:0/1309025056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530801d620 con 0x55620469dc20 2024-04-24T23:48:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.863+0000 7f52fe7fc640 1 -- 172.21.15.107:0/1309025056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530801c600 con 0x55620469dc20 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- 172.21.15.107:0/1309025056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 msgr2=0x55620469c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5308009d40 tx=0x7f5308010800 comp rx=0 tx=0).stop 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- 172.21.15.107:0/1309025056 shutdown_connections 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5562046a2d90 0x5562046ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55620469dc20 0x55620469c400 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 --2- 172.21.15.107:0/1309025056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x55620469bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- 172.21.15.107:0/1309025056 >> 172.21.15.107:0/1309025056 conn(0x5562046988a0 msgr2=0x55620469b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- 172.21.15.107:0/1309025056 shutdown_connections 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- 172.21.15.107:0/1309025056 wait complete. 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 -- start start 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.864+0000 7f531dbde740 1 Processor -- start 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046abb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53141adbc0 con 0x5562046a2d90 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f53141ad4f0 con 0x55620469d230 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f53141aa730 con 0x55620469dc20 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 10 client.0 _ll_register_callbacks cb 0x7f530000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54800/0 (socket says 172.21.15.107:54800) 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 -- 172.21.15.107:0/4055722310 learned_addr learned my addr 172.21.15.107:0/4055722310 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5318c6e640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 -- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 msgr2=0x5562046ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046ac0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 -- 172.21.15.107:0/4055722310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 msgr2=0x5562046abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 -- 172.21.15.107:0/4055722310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562046ad6d0 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f5313fff640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f530000c870 tx=0x7f5300014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e051bcf6ab5eafc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f52f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300018070 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5300005230 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f530801d620 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5300020040 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 10 client.16353 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5562046ae9d0 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.865+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530801d620 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5300029070 con 0x5562046a2d90 2024-04-24T23:48:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 10 client.16353 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 10 client.16353 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 20 client.16353 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562047198e0 0x55620471bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 4 client.16353 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 --> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556204608f00 con 0x5562047198e0 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f5318c6e640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562047198e0 0x55620471bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f52f3fff640 10 client.16353 ms_handle_connect on v2:172.21.15.195:6836/1315131786 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f5318c6e640 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562047198e0 0x55620471bd00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a0b6de6608ad2d87 server_cookie=28542d97bbaa60f5 in_seq=0 out_seq=0 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f52f3fff640 1 -- 172.21.15.107:0/4055722310 <== mds.3 v2:172.21.15.195:6836/1315131786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556204608f00 con 0x5562047198e0 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f52f3fff640 10 client.16353 handle_command_reply: tid=0 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 1 client.16353 shutdown 2024-04-24T23:48:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 msgr2=0x5562046ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.866+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f530000c870 tx=0x7f5300014040 comp rx=0 tx=0).stop 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 shutdown_connections 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:6836/1315131786,v1:172.21.15.195:6837/1315131786] conn(0x5562047198e0 0x55620471bd00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5562046a2d90 0x5562046ad0e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55620469dc20 0x5562046ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 --2- 172.21.15.107:0/4055722310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55620469d230 0x5562046abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f52f3fff640 0 client.16353 ms_handle_reset on v2:172.21.15.195:6836/1315131786 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 >> 172.21.15.107:0/4055722310 conn(0x5562046988a0 msgr2=0x55620469c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 shutdown_connections 2024-04-24T23:48:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 1 -- 172.21.15.107:0/4055722310 wait complete. 2024-04-24T23:48:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531dbde740 20 client.16353 trim_cache size 0 max 16384 2024-04-24T23:48:04.878 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 msgr2=0x7f53040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 0x7f53040a50d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f530c0edde0 tx=0x7f530c0b20f0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 msgr2=0x7f5314074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f530c031ae0 tx=0x7f530c031ee0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 shutdown_connections 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f53040a2c10 0x7f53040a50d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5314103340 0x7f5314074070 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5314102980 0x7f5314073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 --2- 172.21.15.107:0/3544225172 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5314101790 0x7f53140754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 >> 172.21.15.107:0/3544225172 conn(0x7f53140fcd20 msgr2=0x7f5314105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 shutdown_connections 2024-04-24T23:48:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.867+0000 7f531aed2640 1 -- 172.21.15.107:0/3544225172 wait complete. 2024-04-24T23:48:04.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2536111424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 msgr2=0x7fa5cc109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2536111424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc109be0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009ca0 tx=0x7fa5c402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2536111424 shutdown_connections 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2536111424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc109be0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2536111424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 0x7fa5cc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2536111424 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 0x7fa5cc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.982+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2536111424 >> 172.21.15.107:0/2536111424 conn(0x7fa5cc0fcd10 msgr2=0x7fa5cc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2536111424 shutdown_connections 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2536111424 wait complete. 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 Processor -- start 2024-04-24T23:48:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- start start 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 0x7fa5cc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 0x7fa5cc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10d420 con 0x7fa5cc103350 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10d2a0 con 0x7fa5cc102990 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.983+0000 7fa5d0a31640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10d5a0 con 0x7fa5cc1017a0 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54832/0 (socket says 172.21.15.107:54832) 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 -- 172.21.15.107:0/2834559649 learned_addr learned my addr 172.21.15.107:0/2834559649 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 0x7fa5cc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 0x7fa5cc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 -- 172.21.15.107:0/2834559649 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 msgr2=0x7fa5cc197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 0x7fa5cc197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 -- 172.21.15.107:0/2834559649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 msgr2=0x7fa5cc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 0x7fa5cc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 -- 172.21.15.107:0/2834559649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc19f680 con 0x7fa5cc103350 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5cad9d640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4031ae0 tx=0x7fa5c4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5bb7fe640 1 -- 172.21.15.107:0/2834559649 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4003b70 con 0x7fa5cc103350 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc19f910 con 0x7fa5cc103350 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5bb7fe640 1 -- 172.21.15.107:0/2834559649 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c4003d10 con 0x7fa5cc103350 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5bb7fe640 1 -- 172.21.15.107:0/2834559649 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c402fc40 con 0x7fa5cc103350 2024-04-24T23:48:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.984+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc19feb0 con 0x7fa5cc103350 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5bb7fe640 1 -- 172.21.15.107:0/2834559649 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa5c4004050 con 0x7fa5cc103350 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 Processor -- start 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 -- start start 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0230 0x5609517a0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0c20 0x5609517a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560951704670 con 0x5609517a5d90 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5609516ac970 con 0x5609517a0c20 2024-04-24T23:48:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5d3819740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x560951700630 con 0x5609517a0230 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54848/0 (socket says 172.21.15.107:54848) 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1554888085 learned_addr learned my addr 172.21.15.107:0/1554888085 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0c20 0x5609517a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0230 0x5609517a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.987+0000 7fa5bb7fe640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 0x7fa5b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1554888085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0230 msgr2=0x5609517a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0230 0x5609517a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1554888085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0c20 msgr2=0x5609517a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0c20 0x5609517a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1554888085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609516d0400 con 0x5609517a5d90 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5cad9d640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 0x7fa5b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5bb7fe640 1 -- 172.21.15.107:0/2834559649 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa5c40f0490 con 0x7fa5cc103350 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00058c0 tx=0x7fa5c0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7653fad12dd5c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5b8ff9640 1 -- 172.21.15.107:0/1554888085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0004020 con 0x5609517a5d90 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5b8ff9640 1 -- 172.21.15.107:0/1554888085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0005500 con 0x5609517a5d90 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.988+0000 7fa5b8ff9640 1 -- 172.21.15.107:0/1554888085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0004da0 con 0x5609517a5d90 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- 172.21.15.107:0/1554888085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 msgr2=0x5609517a8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5cad9d640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 0x7fa5b40a5050 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40b8ea0 tx=0x7fa5c408bae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00058c0 tx=0x7fa5c0005850 comp rx=0 tx=0).stop 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- 172.21.15.107:0/1554888085 shutdown_connections 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517a8220 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0c20 0x5609517a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1554888085 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0230 0x5609517a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- 172.21.15.107:0/1554888085 >> 172.21.15.107:0/1554888085 conn(0x56095179b8a0 msgr2=0x56095179bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- 172.21.15.107:0/1554888085 shutdown_connections 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- 172.21.15.107:0/1554888085 wait complete. 2024-04-24T23:48:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- start start 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 Processor -- start 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x5609517b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5609517a8d40 con 0x5609517a0230 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5609516c0050 con 0x5609517a5d90 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x5609517a8ec0 con 0x5609517a0c20 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.989+0000 7fa5d3819740 10 client.0 _ll_register_callbacks cb 0x560951417be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54862/0 (socket says 172.21.15.107:54862) 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1906084997 learned_addr learned my addr 172.21.15.107:0/1906084997 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:04.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x5609517b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1906084997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 msgr2=0x5609517aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 msgr2=0x5609517b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x5609517b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1906084997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560951691dc0 con 0x5609517a0230 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0004fd0 tx=0x7fa5c0009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ad05e024090f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c001e470 con 0x5609517a0230 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa5c001e610 con 0x5609517a0230 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c0005500 con 0x5609517a0230 2024-04-24T23:48:04.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5c0031050 con 0x5609517a0230 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5d3819740 10 client.16365 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609514183f0 con 0x5609517a0230 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.990+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0005500 con 0x5609517a0230 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa5c002c850 con 0x5609517a0230 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 10 client.16365 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 10 client.16365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 20 client.16365 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x5609517b5f70 0x5609517b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 4 client.16365 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560951711bb0 con 0x5609517b5f70 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x5609517b5f70 0x5609517b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5b1ffb640 10 client.16365 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:48:04.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.991+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x5609517b5f70 0x5609517b83b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b363b99c62d181f0 server_cookie=56fbf27791e89cdd in_seq=0 out_seq=0 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/1906084997 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560951711bb0 con 0x5609517b5f70 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5b1ffb640 10 client.16365 handle_command_reply: tid=0 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 client.16365 shutdown 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 msgr2=0x5609517aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0004fd0 tx=0x7fa5c0009f70 comp rx=0 tx=0).stop 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 shutdown_connections 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x5609517b5f70 0x5609517b83b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x5609517b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5b1ffb640 0 client.16365 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1906084997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x5609517aa390 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 >> 172.21.15.107:0/1906084997 conn(0x56095179b8a0 msgr2=0x56095179ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 shutdown_connections 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.992+0000 7fa5d3819740 1 -- 172.21.15.107:0/1906084997 wait complete. 2024-04-24T23:48:05.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:04.993+0000 7fa5d3819740 20 client.16365 trim_cache size 0 max 16384 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 Processor -- start 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 -- start start 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x56095179ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1aa480 con 0x5609517a0230 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1abfc0 con 0x5609517a5d90 2024-04-24T23:48:05.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5d3819740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1aab50 con 0x5609517a0c20 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54866/0 (socket says 172.21.15.107:54866) 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1574643933 learned_addr learned my addr 172.21.15.107:0/1574643933 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1574643933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 msgr2=0x5609517b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1574643933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 msgr2=0x56095179ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x56095179ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/1574643933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609514e27d0 con 0x5609517a0230 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c00092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada52289b26b178d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5b3fff640 1 -- 172.21.15.107:0/1574643933 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0039020 con 0x5609517a0230 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5b3fff640 1 -- 172.21.15.107:0/1574643933 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0022040 con 0x5609517a0230 2024-04-24T23:48:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.006+0000 7fa5b3fff640 1 -- 172.21.15.107:0/1574643933 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0005510 con 0x5609517a0230 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- 172.21.15.107:0/1574643933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 msgr2=0x56095179e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c00092a0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- 172.21.15.107:0/1574643933 shutdown_connections 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a5d90 0x56095179ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- 172.21.15.107:0/1574643933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a0230 0x56095179e7b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- 172.21.15.107:0/1574643933 >> 172.21.15.107:0/1574643933 conn(0x56095179b8a0 msgr2=0x56095179ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- 172.21.15.107:0/1574643933 shutdown_connections 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- 172.21.15.107:0/1574643933 wait complete. 2024-04-24T23:48:05.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- start start 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 Processor -- start 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0230 0x5609517aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.007+0000 7fa5d3819740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ab8f0 con 0x5609517a5d90 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1ab220 con 0x5609517a0230 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a8460 con 0x5609517a0c20 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 10 client.0 _ll_register_callbacks cb 0x7fa5bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54882/0 (socket says 172.21.15.107:54882) 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/3920679515 learned_addr learned my addr 172.21.15.107:0/3920679515 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0230 0x5609517aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/3920679515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 msgr2=0x5609517af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0230 msgr2=0x5609517aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0230 0x5609517aeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 -- 172.21.15.107:0/3920679515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609517b06b0 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5c9d9b640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa5c002cd90 tx=0x7fa5c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf846c151f99105b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0039030 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa5c00054b0 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c0022040 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5c002c3e0 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 10 client.16371 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609517b19b0 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.008+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0022040 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa5c0046020 con 0x5609517a5d90 2024-04-24T23:48:05.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 10 client.16371 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 10 client.16371 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 20 client.16371 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x56095181ca20 0x56095181ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 4 client.16371 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 --> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56095170bf00 con 0x56095181ca20 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x56095181ca20 0x56095181ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5b1ffb640 10 client.16371 ms_handle_connect on v2:172.21.15.170:6836/293399389 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.009+0000 7fa5ca59c640 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x56095181ca20 0x56095181ee40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3987e515ab8338ac server_cookie=690e7280c2c68ea6 in_seq=0 out_seq=0 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5b1ffb640 1 -- 172.21.15.107:0/3920679515 <== mds.2 v2:172.21.15.170:6836/293399389 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56095170bf00 con 0x56095181ca20 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5b1ffb640 10 client.16371 handle_command_reply: tid=0 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 client.16371 shutdown 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 msgr2=0x5609517b00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa5c002cd90 tx=0x7fa5c0005f90 comp rx=0 tx=0).stop 2024-04-24T23:48:05.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 shutdown_connections 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:6836/293399389,v1:172.21.15.170:6837/293399389] conn(0x56095181ca20 0x56095181ee40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609517a5d90 0x5609517b00c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x5609517a0c20 0x5609517af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5b1ffb640 0 client.16371 ms_handle_reset on v2:172.21.15.170:6836/293399389 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 --2- 172.21.15.107:0/3920679515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5609517a0230 0x5609517aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 >> 172.21.15.107:0/3920679515 conn(0x56095179b8a0 msgr2=0x5609517a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 shutdown_connections 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 1 -- 172.21.15.107:0/3920679515 wait complete. 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d3819740 20 client.16371 trim_cache size 0 max 16384 2024-04-24T23:48:05.016 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 msgr2=0x7fa5b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 0x7fa5b40a5050 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40b8ea0 tx=0x7fa5c408bae0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 msgr2=0x7fa5cc19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4031ae0 tx=0x7fa5c4031ee0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 shutdown_connections 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5b40a2b90 0x7fa5b40a5050 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5cc103350 0x7fa5cc19f010 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5cc102990 0x7fa5cc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 --2- 172.21.15.107:0/2834559649 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5cc1017a0 0x7fa5cc197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 >> 172.21.15.107:0/2834559649 conn(0x7fa5cc0fcd10 msgr2=0x7fa5cc105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 shutdown_connections 2024-04-24T23:48:05.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.010+0000 7fa5d0a31640 1 -- 172.21.15.107:0/2834559649 wait complete. 2024-04-24T23:48:05.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T23:48:05.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 -- 172.21.15.107:0/1968478726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf81006a0 msgr2=0x7fdaf8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1968478726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf81006a0 0x7fdaf8100ac0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0009ca0 tx=0x7fdaf002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 -- 172.21.15.107:0/1968478726 shutdown_connections 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1968478726 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf8102250 0x7fdaf8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1968478726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 0x7fdaf8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1968478726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf81006a0 0x7fdaf8100ac0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.120+0000 7fdafec9e640 1 -- 172.21.15.107:0/1968478726 >> 172.21.15.107:0/1968478726 conn(0x7fdaf80fbc10 msgr2=0x7fdaf80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- 172.21.15.107:0/1968478726 shutdown_connections 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- 172.21.15.107:0/1968478726 wait complete. 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 Processor -- start 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- start start 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf81006a0 0x7fdaf806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 0x7fdaf819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf810c3c0 con 0x7fdaf8102250 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf810c240 con 0x7fdaf8101890 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafec9e640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf810c540 con 0x7fdaf81006a0 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafd23b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafd23b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54892/0 (socket says 172.21.15.107:54892) 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafd23b640 1 -- 172.21.15.107:0/1511960078 learned_addr learned my addr 172.21.15.107:0/1511960078 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.121+0000 7fdafca3a640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf81006a0 0x7fdaf806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 0x7fdaf819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 -- 172.21.15.107:0/1511960078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf81006a0 msgr2=0x7fdaf806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf81006a0 0x7fdaf806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 -- 172.21.15.107:0/1511960078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 msgr2=0x7fdaf819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 0x7fdaf819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 -- 172.21.15.107:0/1511960078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf819f750 con 0x7fdaf8102250 2024-04-24T23:48:05.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafd23b640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009fd0 tx=0x7fdae800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdaf5ffb640 1 -- 172.21.15.107:0/1511960078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8011830 con 0x7fdaf8102250 2024-04-24T23:48:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdaf5ffb640 1 -- 172.21.15.107:0/1511960078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae8016070 con 0x7fdaf8102250 2024-04-24T23:48:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf819fa40 con 0x7fdaf8102250 2024-04-24T23:48:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdaf5ffb640 1 -- 172.21.15.107:0/1511960078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae800f2e0 con 0x7fdaf8102250 2024-04-24T23:48:05.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.122+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf81a0010 con 0x7fdaf8102250 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.124+0000 7fdaf5ffb640 1 -- 172.21.15.107:0/1511960078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdae80119d0 con 0x7fdaf8102250 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 Processor -- start 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 -- start start 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d065b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a8cfc0670 con 0x561a8d060640 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a8cf68970 con 0x561a8d0661a0 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdb019ac740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a8cfbc630 con 0x561a8d061030 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf5ffb640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 0x7fdae40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdafd23b640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 0x7fdae40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54894/0 (socket says 172.21.15.107:54894) 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 -- 172.21.15.107:0/61597843 learned_addr learned my addr 172.21.15.107:0/61597843 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdafca3a640 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d065b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf5ffb640 1 -- 172.21.15.107:0/1511960078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdae80ce980 con 0x7fdaf8102250 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 -- 172.21.15.107:0/61597843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 msgr2=0x561a8d065b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d065b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 -- 172.21.15.107:0/61597843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 msgr2=0x561a8d068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d068630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 -- 172.21.15.107:0/61597843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a8cf8c400 con 0x561a8d060640 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdafd23b640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 0x7fdae40a5050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdae8097270 tx=0x7fdae801b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdae0005cf0 tx=0x7fdae000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e005d332bcc59c06 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.125+0000 7fdaeeffd640 1 -- 172.21.15.107:0/61597843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0004030 con 0x561a8d060640 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdaeeffd640 1 -- 172.21.15.107:0/61597843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae0005500 con 0x561a8d060640 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdaeeffd640 1 -- 172.21.15.107:0/61597843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0004c00 con 0x561a8d060640 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 -- 172.21.15.107:0/61597843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 msgr2=0x561a8d060a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdae0005cf0 tx=0x7fdae000b840 comp rx=0 tx=0).stop 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 -- 172.21.15.107:0/61597843 shutdown_connections 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d065b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 --2- 172.21.15.107:0/61597843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d060a60 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 -- 172.21.15.107:0/61597843 >> 172.21.15.107:0/61597843 conn(0x561a8d0578a0 msgr2=0x561a8d057ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 -- 172.21.15.107:0/61597843 shutdown_connections 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.126+0000 7fdb019ac740 1 -- 172.21.15.107:0/61597843 wait complete. 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 -- start start 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 Processor -- start 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561a8d069150 con 0x561a8d060640 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561a8cf7c050 con 0x561a8d0661a0 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x561a8d0692d0 con 0x561a8d061030 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdb019ac740 10 client.0 _ll_register_callbacks cb 0x561a8ccd3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54904/0 (socket says 172.21.15.107:54904) 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 -- 172.21.15.107:0/3114883909 learned_addr learned my addr 172.21.15.107:0/3114883909 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdafca3a640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 -- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 msgr2=0x561a8d06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 -- 172.21.15.107:0/3114883909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 msgr2=0x561a8d071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdaf7fff640 1 -- 172.21.15.107:0/3114883909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a8cf4ddc0 con 0x561a8d060640 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.127+0000 7fdafca3a640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdae0006010 tx=0x7fdae0012300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec7f91afa4aaa48 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0028060 con 0x561a8d060640 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdae0004940 con 0x561a8d060640 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae0005500 con 0x561a8d060640 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdae000ba90 con 0x561a8d060640 2024-04-24T23:48:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 10 client.16383 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a8ccd43f0 con 0x561a8d060640 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae0005500 con 0x561a8d060640 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdae0010c00 con 0x561a8d060640 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 10 client.16383 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 10 client.16383 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 20 client.16383 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d076300 0x561a8d078710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 4 client.16383 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.128+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a8cfcdbb0 con 0x561a8d076300 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.129+0000 7fdafca3a640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d076300 0x561a8d078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.129+0000 7fdad7fff640 10 client.16383 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.129+0000 7fdafca3a640 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d076300 0x561a8d078710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae964d51e1060ffa server_cookie=5c8d8632ac964947 in_seq=0 out_seq=0 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdad7fff640 1 -- 172.21.15.107:0/3114883909 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a8cfcdbb0 con 0x561a8d076300 2024-04-24T23:48:05.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdad7fff640 10 client.16383 handle_command_reply: tid=0 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 client.16383 shutdown 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 msgr2=0x561a8d06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdae0006010 tx=0x7fdae0012300 comp rx=0 tx=0).stop 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 shutdown_connections 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d076300 0x561a8d078710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d071cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdad7fff640 0 client.16383 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d061030 0x561a8d06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 --2- 172.21.15.107:0/3114883909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d06a730 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 >> 172.21.15.107:0/3114883909 conn(0x561a8d0578a0 msgr2=0x561a8d056b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 shutdown_connections 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 1 -- 172.21.15.107:0/3114883909 wait complete. 2024-04-24T23:48:05.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.130+0000 7fdb019ac740 20 client.16383 trim_cache size 0 max 16384 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 Processor -- start 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 -- start start 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 0x561a8d071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d0661a0 0x561a8d061c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a83c0 con 0x561a8d060640 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a9f00 con 0x561a8d061030 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdb019ac740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a8a90 con 0x561a8d0661a0 2024-04-24T23:48:05.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54914/0 (socket says 172.21.15.107:54914) 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 -- 172.21.15.107:0/2991989309 learned_addr learned my addr 172.21.15.107:0/2991989309 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdafca3a640 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 0x561a8d071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 -- 172.21.15.107:0/2991989309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d0661a0 msgr2=0x561a8d061c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d0661a0 0x561a8d061c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 -- 172.21.15.107:0/2991989309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 msgr2=0x561a8d071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 0x561a8d071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 -- 172.21.15.107:0/2991989309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a8cd9e7d0 con 0x561a8d060640 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdafca3a640 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 0x561a8d071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdae0001eb0 tx=0x7fdae0023570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ecb1b6575f8d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaee7fc640 1 -- 172.21.15.107:0/2991989309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0028070 con 0x561a8d060640 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaee7fc640 1 -- 172.21.15.107:0/2991989309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae000bcf0 con 0x561a8d060640 2024-04-24T23:48:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.144+0000 7fdaee7fc640 1 -- 172.21.15.107:0/2991989309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0004ab0 con 0x561a8d060640 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- 172.21.15.107:0/2991989309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 msgr2=0x561a8d061720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdae0001eb0 tx=0x7fdae0023570 comp rx=0 tx=0).stop 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- 172.21.15.107:0/2991989309 shutdown_connections 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d0661a0 0x561a8d061c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d061030 0x561a8d071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 --2- 172.21.15.107:0/2991989309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d060640 0x561a8d061720 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- 172.21.15.107:0/2991989309 >> 172.21.15.107:0/2991989309 conn(0x561a8d0578a0 msgr2=0x561a8d05ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- 172.21.15.107:0/2991989309 shutdown_connections 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- 172.21.15.107:0/2991989309 wait complete. 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 -- start start 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.145+0000 7fdb019ac740 1 Processor -- start 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 0x561a8d06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d0704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a9830 con 0x561a8d061030 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a9160 con 0x561a8d0661a0 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf81a63a0 con 0x561a8d060640 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 10 client.0 _ll_register_callbacks cb 0x7fdaf0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54918/0 (socket says 172.21.15.107:54918) 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 -- 172.21.15.107:0/1211661520 learned_addr learned my addr 172.21.15.107:0/1211661520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 0x561a8d06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 -- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 msgr2=0x561a8d06efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 0x561a8d06efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 -- 172.21.15.107:0/1211661520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 msgr2=0x561a8d0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d0704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 -- 172.21.15.107:0/1211661520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a8d070ad0 con 0x561a8d061030 2024-04-24T23:48:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 0x561a8d06efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdafca3a640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdaf002ae80 tx=0x7fdaf0031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8185e18b731fa394 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdad7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf004b020 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdaf0003b70 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae000bcf0 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdaf0003d70 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 10 client.16389 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.146+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a8d071de0 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae000bcf0 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fdaf002f610 con 0x561a8d061030 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 10 client.16389 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 10 client.16389 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 20 client.16389 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d0dcf80 0x561a8d0df3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 4 client.16389 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 --> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a8cfc7f00 con 0x561a8d0dcf80 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.147+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d0dcf80 0x561a8d0df3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdaf7fff640 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d0dcf80 0x561a8d0df3a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36ff1418ed0555f0 server_cookie=a0422c0f2faf1ae9 in_seq=0 out_seq=0 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdad7fff640 10 client.16389 ms_handle_connect on v2:172.21.15.195:6838/4197811615 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdad7fff640 1 -- 172.21.15.107:0/1211661520 <== mds.1 v2:172.21.15.195:6838/4197811615 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a8cfc7f00 con 0x561a8d0dcf80 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdad7fff640 10 client.16389 handle_command_reply: tid=0 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 client.16389 shutdown 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 msgr2=0x561a8d06f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdaf002ae80 tx=0x7fdaf0031650 comp rx=0 tx=0).stop 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 shutdown_connections 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:6838/4197811615,v1:172.21.15.195:6839/4197811615] conn(0x561a8d0dcf80 0x561a8d0df3a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561a8d0661a0 0x561a8d0704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561a8d061030 0x561a8d06f520 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdad7fff640 0 client.16389 ms_handle_reset on v2:172.21.15.195:6838/4197811615 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 --2- 172.21.15.107:0/1211661520 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x561a8d060640 0x561a8d06efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 >> 172.21.15.107:0/1211661520 conn(0x561a8d0578a0 msgr2=0x561a8d061720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 shutdown_connections 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 1 -- 172.21.15.107:0/1211661520 wait complete. 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.148+0000 7fdb019ac740 20 client.16389 trim_cache size 0 max 16384 2024-04-24T23:48:05.155 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 msgr2=0x7fdae40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 0x7fdae40a5050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdae8097270 tx=0x7fdae801b000 comp rx=0 tx=0).stop 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 msgr2=0x7fdaf819ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009fd0 tx=0x7fdae800b8e0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 shutdown_connections 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fdae40a2b90 0x7fdae40a5050 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaf8102250 0x7fdaf819ee90 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaf8101890 0x7fdaf819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 --2- 172.21.15.107:0/1511960078 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdaf81006a0 0x7fdaf806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 >> 172.21.15.107:0/1511960078 conn(0x7fdaf80fbc10 msgr2=0x7fdaf8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 shutdown_connections 2024-04-24T23:48:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.149+0000 7fdafec9e640 1 -- 172.21.15.107:0/1511960078 wait complete. 2024-04-24T23:48:05.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 -- 172.21.15.107:0/2073380645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28107500 msgr2=0x7f1d281098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 --2- 172.21.15.107:0/2073380645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28107500 0x7f1d281098f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ca0 tx=0x7f1d2002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 -- 172.21.15.107:0/2073380645 shutdown_connections 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 --2- 172.21.15.107:0/2073380645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28107500 0x7f1d281098f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 --2- 172.21.15.107:0/2073380645 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d28104bd0 0x7f1d28106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 --2- 172.21.15.107:0/2073380645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 -- 172.21.15.107:0/2073380645 >> 172.21.15.107:0/2073380645 conn(0x7f1d280fbc10 msgr2=0x7f1d280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 -- 172.21.15.107:0/2073380645 shutdown_connections 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.260+0000 7f1d2f693640 1 -- 172.21.15.107:0/2073380645 wait complete. 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 Processor -- start 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 -- start start 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d28107500 0x7f1d28100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d440 con 0x7f1d28104bd0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d2c0 con 0x7f1d281022a0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2810d5c0 con 0x7f1d28107500 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54942/0 (socket says 172.21.15.107:54942) 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/3789318387 learned_addr learned my addr 172.21.15.107:0/3789318387 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/3789318387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d28107500 msgr2=0x7f1d28100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d28107500 0x7f1d28100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/3789318387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 msgr2=0x7f1d28101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/3789318387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d28101020 con 0x7f1d28104bd0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800d740 tx=0x7f1d1800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d1e7fc640 1 -- 172.21.15.107:0/3789318387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18017070 con 0x7f1d28104bd0 2024-04-24T23:48:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d281a1bd0 con 0x7f1d28104bd0 2024-04-24T23:48:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.261+0000 7f1d1e7fc640 1 -- 172.21.15.107:0/3789318387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1800de50 con 0x7f1d28104bd0 2024-04-24T23:48:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.262+0000 7f1d1e7fc640 1 -- 172.21.15.107:0/3789318387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18004e00 con 0x7f1d28104bd0 2024-04-24T23:48:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.262+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d281a2120 con 0x7f1d28104bd0 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d1e7fc640 1 -- 172.21.15.107:0/3789318387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1d18004fa0 con 0x7f1d28104bd0 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 Processor -- start 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 -- start start 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf13720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf16220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cd2ce72670 con 0x55cd2cf0e230 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55cd2ce1a970 con 0x55cd2cf0ec20 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d3239f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55cd2ce6e630 con 0x55cd2cf13d90 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d2dc30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d2dc30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54958/0 (socket says 172.21.15.107:54958) 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d2dc30640 1 -- 172.21.15.107:0/2231909667 learned_addr learned my addr 172.21.15.107:0/2231909667 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.264+0000 7f1d1e7fc640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 0x7f1d140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf16220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 0x7f1d140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf13720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 -- 172.21.15.107:0/2231909667 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 msgr2=0x55cd2cf16220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf16220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 -- 172.21.15.107:0/2231909667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 msgr2=0x55cd2cf13720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf13720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 -- 172.21.15.107:0/2231909667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd2ce3e400 con 0x55cd2cf0e230 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf16220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d1e7fc640 1 -- 172.21.15.107:0/3789318387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1d180cee90 con 0x7f1d28104bd0 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1d20031c10 tx=0x7f1d20031eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2b69aeefa280c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d137fe640 1 -- 172.21.15.107:0/2231909667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2004f070 con 0x55cd2cf0e230 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d137fe640 1 -- 172.21.15.107:0/2231909667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d20036040 con 0x55cd2cf0e230 2024-04-24T23:48:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d137fe640 1 -- 172.21.15.107:0/2231909667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20030940 con 0x55cd2cf0e230 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.265+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 0x7f1d140a5050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d2004c000 tx=0x7f1d2004a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- 172.21.15.107:0/2231909667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 msgr2=0x55cd2cf0e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1d20031c10 tx=0x7f1d20031eb0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- 172.21.15.107:0/2231909667 shutdown_connections 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf16220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf13720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2231909667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0e650 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- 172.21.15.107:0/2231909667 >> 172.21.15.107:0/2231909667 conn(0x55cd2cf098a0 msgr2=0x55cd2cf09ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- 172.21.15.107:0/2231909667 shutdown_connections 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- 172.21.15.107:0/2231909667 wait complete. 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 -- start start 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.266+0000 7f1d3239f740 1 Processor -- start 2024-04-24T23:48:05.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf18230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf18770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cd2cf16bc0 con 0x55cd2cf13d90 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55cd2ce2e050 con 0x55cd2cf0ec20 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x55cd2cf16d40 con 0x55cd2cf0e230 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 10 client.0 _ll_register_callbacks cb 0x55cd2cb85be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54964/0 (socket says 172.21.15.107:54964) 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 -- 172.21.15.107:0/1620291587 learned_addr learned my addr 172.21.15.107:0/1620291587 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf18230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf18770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 -- 172.21.15.107:0/1620291587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 msgr2=0x55cd2cf18230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf18230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 -- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 msgr2=0x55cd2cf18770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf18770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 -- 172.21.15.107:0/1620291587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd2cdffdc0 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf18770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf18230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009bf0 tx=0x7f1d2002f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34afd66335899c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20057020 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1d20030da0 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d20036040 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1d20062050 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 10 client.16401 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.267+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd2cb863f0 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d20036040 con 0x55cd2cf13d90 2024-04-24T23:48:05.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1d20053b00 con 0x55cd2cf13d90 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 10 client.16401 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 20 client.16401 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf23d40 0x55cd2cf261c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 4 client.16401 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd2ce7fbb0 con 0x55cd2cf23d40 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.268+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf23d40 0x55cd2cf261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d10ff9640 10 client.16401 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf23d40 0x55cd2cf261c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff369af2b716fb2a server_cookie=e3536e06676e9b0c in_seq=0 out_seq=0 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d10ff9640 1 -- 172.21.15.107:0/1620291587 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd2ce7fbb0 con 0x55cd2cf23d40 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d10ff9640 10 client.16401 handle_command_reply: tid=0 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 client.16401 shutdown 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 msgr2=0x55cd2cf1f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009bf0 tx=0x7f1d2002f3b0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 shutdown_connections 2024-04-24T23:48:05.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf23d40 0x55cd2cf261c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1f740 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf18770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d10ff9640 0 client.16401 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 --2- 172.21.15.107:0/1620291587 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf18230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.269+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 >> 172.21.15.107:0/1620291587 conn(0x55cd2cf098a0 msgr2=0x55cd2cf08b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.270+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 shutdown_connections 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.270+0000 7f1d3239f740 1 -- 172.21.15.107:0/1620291587 wait complete. 2024-04-24T23:48:05.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.270+0000 7f1d3239f740 20 client.16401 trim_cache size 0 max 16384 2024-04-24T23:48:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 Processor -- start 2024-04-24T23:48:05.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 -- start start 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf0cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281ac6c0 con 0x55cd2cf0e230 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281ae200 con 0x55cd2cf0ec20 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d3239f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281acd90 con 0x55cd2cf13d90 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54972/0 (socket says 172.21.15.107:54972) 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 -- 172.21.15.107:0/239239949 learned_addr learned my addr 172.21.15.107:0/239239949 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2dc30640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf0cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 -- 172.21.15.107:0/239239949 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 msgr2=0x55cd2cf0cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf0cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 -- 172.21.15.107:0/239239949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 msgr2=0x55cd2cf1f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 -- 172.21.15.107:0/239239949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd2cc507d0 con 0x55cd2cf0e230 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.283+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x55cd2cf0f020 tx=0x7f1d24005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a554bce62744a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d12ffd640 1 -- 172.21.15.107:0/239239949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2401d070 con 0x55cd2cf0e230 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d12ffd640 1 -- 172.21.15.107:0/239239949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d24019dc0 con 0x55cd2cf0e230 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d12ffd640 1 -- 172.21.15.107:0/239239949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400fc00 con 0x55cd2cf0e230 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- 172.21.15.107:0/239239949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 msgr2=0x55cd2cf0c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x55cd2cf0f020 tx=0x7f1d24005f10 comp rx=0 tx=0).stop 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- 172.21.15.107:0/239239949 shutdown_connections 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf13d90 0x55cd2cf0cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 --2- 172.21.15.107:0/239239949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0e230 0x55cd2cf0c580 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- 172.21.15.107:0/239239949 >> 172.21.15.107:0/239239949 conn(0x55cd2cf098a0 msgr2=0x55cd2cf08b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- 172.21.15.107:0/239239949 shutdown_connections 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- 172.21.15.107:0/239239949 wait complete. 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.284+0000 7f1d3239f740 1 -- start start 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 Processor -- start 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf1cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281adb30 con 0x55cd2cf0ec20 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281ad460 con 0x55cd2cf13d90 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1d281aa6a0 con 0x55cd2cf0e230 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d3239f740 10 client.0 _ll_register_callbacks cb 0x7f1d2004d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54976/0 (socket says 172.21.15.107:54976) 2024-04-24T23:48:05.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/2555537960 learned_addr learned my addr 172.21.15.107:0/2555537960 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf1cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/2555537960 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 msgr2=0x55cd2cf1cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf1cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf13d90 msgr2=0x55cd2cf1e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 -- 172.21.15.107:0/2555537960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd2cf1e8f0 con 0x55cd2cf0ec20 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d2cc2e640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1d180cc5c0 tx=0x7f1d1806e3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b0c2d056c9bf38 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.285+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18020040 con 0x55cd2cf0ec20 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1d18097990 con 0x55cd2cf0ec20 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d24019dc0 con 0x55cd2cf0ec20 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1d1809f6d0 con 0x55cd2cf0ec20 2024-04-24T23:48:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 10 client.16407 fetch_fsmap learned FSMap version 29 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd2cf8a300 con 0x55cd2cf0ec20 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d24019dc0 con 0x55cd2cf0ec20 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1d18097070 con 0x55cd2cf0ec20 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 10 client.16407 fetch_fsmap finished waiting for FSMap version 29 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 20 client.16407 populate_metadata read hostname 'smithi107' 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf8ad50 0x55cd2cf8d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 4 client.16407 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 --> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd2ce79f00 con 0x55cd2cf8ad50 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.286+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf8ad50 0x55cd2cf8d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d2d42f640 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf8ad50 0x55cd2cf8d170 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d070262f23f87ce1 server_cookie=2b5838a52c7d7e74 in_seq=0 out_seq=0 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d10ff9640 10 client.16407 ms_handle_connect on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d10ff9640 1 -- 172.21.15.107:0/2555537960 <== mds.0 v2:172.21.15.170:6838/2021576737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd2ce79f00 con 0x55cd2cf8ad50 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d10ff9640 10 client.16407 handle_command_reply: tid=0 2024-04-24T23:48:05.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 client.16407 shutdown 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 msgr2=0x55cd2cf1d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1d180cc5c0 tx=0x7f1d1806e3e0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 shutdown_connections 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:6838/2021576737,v1:172.21.15.170:6839/2021576737] conn(0x55cd2cf8ad50 0x55cd2cf8d170 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd2cf13d90 0x55cd2cf1e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cd2cf0ec20 0x55cd2cf1d2f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d10ff9640 0 client.16407 ms_handle_reset on v2:172.21.15.170:6838/2021576737 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 --2- 172.21.15.107:0/2555537960 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x55cd2cf0e230 0x55cd2cf1cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 >> 172.21.15.107:0/2555537960 conn(0x55cd2cf098a0 msgr2=0x55cd2cf0f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 shutdown_connections 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 1 -- 172.21.15.107:0/2555537960 wait complete. 2024-04-24T23:48:05.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.287+0000 7f1d3239f740 20 client.16407 trim_cache size 0 max 16384 2024-04-24T23:48:05.295 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-04-24T23:48:05.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 msgr2=0x7f1d140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 0x7f1d140a5050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d2004c000 tx=0x7f1d2004a000 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 msgr2=0x7f1d28100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800d740 tx=0x7f1d1800dc00 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 shutdown_connections 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1d140a2b90 0x7f1d140a5050 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1d28107500 0x7f1d28100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1d28104bd0 0x7f1d28100250 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 --2- 172.21.15.107:0/3789318387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d281022a0 0x7f1d28101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 >> 172.21.15.107:0/3789318387 conn(0x7f1d280fbc10 msgr2=0x7f1d28102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 shutdown_connections 2024-04-24T23:48:05.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:05.288+0000 7f1d2f693640 1 -- 172.21.15.107:0/3789318387 wait complete. 2024-04-24T23:48:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:05 smithi195 ceph-mon[33019]: pgmap v451: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 42 op/s 2024-04-24T23:48:05.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:05 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2667294597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:05 smithi170 ceph-mon[32714]: pgmap v451: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 42 op/s 2024-04-24T23:48:05.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:05 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2667294597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:05.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:05 smithi107 ceph-mon[29118]: pgmap v451: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 42 op/s 2024-04-24T23:48:05.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:05 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2667294597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:06.300 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-24T23:48:06.301 INFO:tasks.fwd_scrub:done joining 2024-04-24T23:48:06.301 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-24T23:48:06.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3527045489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c28102990 msgr2=0x7f7c28102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3527045489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c28102990 0x7f7c28102e10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009900 tx=0x7f7c1002a410 comp rx=0 tx=0).stop 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3527045489 shutdown_connections 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3527045489 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28103350 0x7f7c28109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3527045489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c28102990 0x7f7c28102e10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3527045489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c281017a0 0x7f7c28101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3527045489 >> 172.21.15.107:0/3527045489 conn(0x7f7c280fccf0 msgr2=0x7f7c280ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3527045489 shutdown_connections 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3527045489 wait complete. 2024-04-24T23:48:06.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 Processor -- start 2024-04-24T23:48:06.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.422+0000 7f7c2db5a640 1 -- start start 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 0x7f7c28197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 0x7f7c2819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d420 con 0x7f7c281017a0 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d2a0 con 0x7f7c28103350 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d5a0 con 0x7f7c28102990 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55002/0 (socket says 172.21.15.107:55002) 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 -- 172.21.15.107:0/3390343234 learned_addr learned my addr 172.21.15.107:0/3390343234 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c1effd640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 0x7f7c28197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c27fff640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 0x7f7c2819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 -- 172.21.15.107:0/3390343234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 msgr2=0x7f7c28197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 0x7f7c28197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 -- 172.21.15.107:0/3390343234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 msgr2=0x7f7c2819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 0x7f7c2819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 -- 172.21.15.107:0/3390343234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2819f6c0 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c1effd640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 0x7f7c28197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c27fff640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 0x7f7c2819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c277fe640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c970 tx=0x7f7c1800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18004220 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c180043c0 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2819f9b0 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800f450 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.423+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2819fed0 con 0x7f7c281017a0 2024-04-24T23:48:06.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.425+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28068560 con 0x7f7c281017a0 2024-04-24T23:48:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.426+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7c180079b0 con 0x7f7c281017a0 2024-04-24T23:48:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.426+0000 7f7c257fa640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 0x7f7c140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.426+0000 7f7c1effd640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 0x7f7c140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.426+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7c180cf250 con 0x7f7c281017a0 2024-04-24T23:48:06.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.427+0000 7f7c1effd640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 0x7f7c140a5050 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7c10000c00 tx=0x7f7c100315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:06.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.431+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7c18014030 con 0x7f7c281017a0 2024-04-24T23:48:06.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.607+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c28108b40 con 0x7f7c281017a0 2024-04-24T23:48:06.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.608+0000 7f7c257fa640 1 -- 172.21.15.107:0/3390343234 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7f7c1809bde0 con 0x7f7c281017a0 2024-04-24T23:48:06.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:06.608 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:06.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.610+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 msgr2=0x7f7c140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.610+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 0x7f7c140a5050 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7c10000c00 tx=0x7f7c100315d0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.610+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 msgr2=0x7f7c28197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.610+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c970 tx=0x7f7c1800ce30 comp rx=0 tx=0).stop 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 shutdown_connections 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f7c140a2b90 0x7f7c140a5050 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c28103350 0x7f7c2819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f7c28102990 0x7f7c28197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 --2- 172.21.15.107:0/3390343234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c281017a0 0x7f7c28197aa0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 >> 172.21.15.107:0/3390343234 conn(0x7f7c280fccf0 msgr2=0x7f7c28105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 shutdown_connections 2024-04-24T23:48:06.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.611+0000 7f7c2db5a640 1 -- 172.21.15.107:0/3390343234 wait complete. 2024-04-24T23:48:06.623 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:48:06.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/624394116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88102980 msgr2=0x7ffa88102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/624394116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88102980 0x7ffa88102e00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009900 tx=0x7ffa7c02a410 comp rx=0 tx=0).stop 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/624394116 shutdown_connections 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/624394116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 0x7ffa88109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/624394116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88102980 0x7ffa88102e00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/624394116 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88101790 0x7ffa88101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/624394116 >> 172.21.15.107:0/624394116 conn(0x7ffa880fcd20 msgr2=0x7ffa880ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:06.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/624394116 shutdown_connections 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.725+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/624394116 wait complete. 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 Processor -- start 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 -- start start 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88102980 0x7ffa88198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 0x7ffa8819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810d3f0 con 0x7ffa88101790 2024-04-24T23:48:06.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810d270 con 0x7ffa88103340 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8e2b3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810d570 con 0x7ffa88102980 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37428/0 (socket says 172.21.15.107:37428) 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa87fff640 1 -- 172.21.15.107:0/1376128425 learned_addr learned my addr 172.21.15.107:0/1376128425 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.726+0000 7ffa8c850640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 0x7ffa8819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 -- 172.21.15.107:0/1376128425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88102980 msgr2=0x7ffa88198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa877fe640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88102980 0x7ffa88198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88102980 0x7ffa88198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 -- 172.21.15.107:0/1376128425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 msgr2=0x7ffa8819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 0x7ffa8819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 -- 172.21.15.107:0/1376128425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8819f6e0 con 0x7ffa88101790 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa87fff640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800c970 tx=0x7ffa7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78004220 con 0x7ffa88101790 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8819f9d0 con 0x7ffa88101790 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa780043c0 con 0x7ffa88101790 2024-04-24T23:48:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800f450 con 0x7ffa88101790 2024-04-24T23:48:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.727+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8819ff20 con 0x7ffa88101790 2024-04-24T23:48:06.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.728+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa88108b50 con 0x7ffa88101790 2024-04-24T23:48:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.729+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ffa780079b0 con 0x7ffa88101790 2024-04-24T23:48:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.730+0000 7ffa857fa640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.730+0000 7ffa877fe640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.730+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffa780cf1a0 con 0x7ffa88101790 2024-04-24T23:48:06.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.730+0000 7ffa877fe640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 0x7ffa700a5050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c031610 tx=0x7ffa7c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:06.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.734+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa78014030 con 0x7ffa88101790 2024-04-24T23:48:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.908+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa8802d3f0 con 0x7ffa88101790 2024-04-24T23:48:06.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.909+0000 7ffa857fa640 1 -- 172.21.15.107:0/1376128425 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7ffa7809bd30 con 0x7ffa88101790 2024-04-24T23:48:06.909 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:48:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 msgr2=0x7ffa700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 0x7ffa700a5050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c031610 tx=0x7ffa7c000bc0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 msgr2=0x7ffa88197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800c970 tx=0x7ffa7800ce30 comp rx=0 tx=0).stop 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 shutdown_connections 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa88103340 0x7ffa8819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7ffa88102980 0x7ffa88198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 --2- 172.21.15.107:0/1376128425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa88101790 0x7ffa88197ac0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 >> 172.21.15.107:0/1376128425 conn(0x7ffa880fcd20 msgr2=0x7ffa88105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 shutdown_connections 2024-04-24T23:48:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:06.911+0000 7ffa8e2b3640 1 -- 172.21.15.107:0/1376128425 wait complete. 2024-04-24T23:48:06.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:48:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- 172.21.15.107:0/3641088479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 msgr2=0x7f41b0101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- 172.21.15.107:0/3641088479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b0101cf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4198009900 tx=0x7f419802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- 172.21.15.107:0/3641088479 shutdown_connections 2024-04-24T23:48:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- 172.21.15.107:0/3641088479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0102230 0x7f41b0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- 172.21.15.107:0/3641088479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b0101cf0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- 172.21.15.107:0/3641088479 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0100680 0x7f41b0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- 172.21.15.107:0/3641088479 >> 172.21.15.107:0/3641088479 conn(0x7f41b00fbbf0 msgr2=0x7f41b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- 172.21.15.107:0/3641088479 shutdown_connections 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- 172.21.15.107:0/3641088479 wait complete. 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 Processor -- start 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- start start 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 0x7f41b0069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 0x7f41b019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010c2e0 con 0x7f41b0101870 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010c160 con 0x7f41b0100680 2024-04-24T23:48:07.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.024+0000 7f41b5c27640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010c460 con 0x7f41b0102230 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37454/0 (socket says 172.21.15.107:37454) 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 -- 172.21.15.107:0/1727023170 learned_addr learned my addr 172.21.15.107:0/1727023170 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41af7fe640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 0x7f41b0069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41affff640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 0x7f41b019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 -- 172.21.15.107:0/1727023170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 msgr2=0x7f41b019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 0x7f41b019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 -- 172.21.15.107:0/1727023170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 msgr2=0x7f41b0069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 0x7f41b0069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 -- 172.21.15.107:0/1727023170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b019f680 con 0x7f41b0101870 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41affff640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 0x7f41b019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41af7fe640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 0x7f41b0069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41a7fff640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4198009230 tx=0x7f4198004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419803f030 con 0x7f41b0101870 2024-04-24T23:48:07.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4198003ce0 con 0x7f41b0101870 2024-04-24T23:48:07.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b019f910 con 0x7f41b0101870 2024-04-24T23:48:07.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198035070 con 0x7f41b0101870 2024-04-24T23:48:07.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.025+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b019fe00 con 0x7f41b0101870 2024-04-24T23:48:07.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.026+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b01079c0 con 0x7f41b0101870 2024-04-24T23:48:07.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.027+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4198042030 con 0x7f41b0101870 2024-04-24T23:48:07.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.028+0000 7f41ad7fa640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 0x7f419c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.028+0000 7f41af7fe640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 0x7f419c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.028+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f41980efc30 con 0x7f41b0101870 2024-04-24T23:48:07.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.028+0000 7f41af7fe640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 0x7f419c0a5050 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f41b01016d0 tx=0x7f41a000a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.032+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41980bc7c0 con 0x7f41b0101870 2024-04-24T23:48:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.208+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41b00623e0 con 0x7f41b0101870 2024-04-24T23:48:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.209+0000 7f41ad7fa640 1 -- 172.21.15.107:0/1727023170 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f41980bc0c0 con 0x7f41b0101870 2024-04-24T23:48:07.209 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 msgr2=0x7f419c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 0x7f419c0a5050 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f41b01016d0 tx=0x7f41a000a5e0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 msgr2=0x7f41b019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4198009230 tx=0x7f4198004430 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 shutdown_connections 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f419c0a2b90 0x7f419c0a5050 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f41b0102230 0x7f41b019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41b0101870 0x7f41b019c780 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 --2- 172.21.15.107:0/1727023170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b0100680 0x7f41b0069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 >> 172.21.15.107:0/1727023170 conn(0x7f41b00fbbf0 msgr2=0x7f41b0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 shutdown_connections 2024-04-24T23:48:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.211+0000 7f41b5c27640 1 -- 172.21.15.107:0/1727023170 wait complete. 2024-04-24T23:48:07.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:48:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 -- 172.21.15.107:0/3837003472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 msgr2=0x7fd5a0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/3837003472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a0100aa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd594009900 tx=0x7fd59402a410 comp rx=0 tx=0).stop 2024-04-24T23:48:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 -- 172.21.15.107:0/3837003472 shutdown_connections 2024-04-24T23:48:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/3837003472 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0102230 0x7fd5a0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/3837003472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0101870 0x7fd5a0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/3837003472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a0100aa0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 -- 172.21.15.107:0/3837003472 >> 172.21.15.107:0/3837003472 conn(0x7fd5a00fbbf0 msgr2=0x7fd5a00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 -- 172.21.15.107:0/3837003472 shutdown_connections 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 -- 172.21.15.107:0/3837003472 wait complete. 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.324+0000 7fd5a6eee640 1 Processor -- start 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 -- start start 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 0x7fd5a0196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 0x7fd5a019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010c350 con 0x7fd5a0100680 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010c1d0 con 0x7fd5a0102230 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010c4d0 con 0x7fd5a0101870 2024-04-24T23:48:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37486/0 (socket says 172.21.15.107:37486) 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 -- 172.21.15.107:0/1261158412 learned_addr learned my addr 172.21.15.107:0/1261158412 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a548b640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 0x7fd5a019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd59ffff640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 0x7fd5a0196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 -- 172.21.15.107:0/1261158412 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 msgr2=0x7fd5a0196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 0x7fd5a0196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 -- 172.21.15.107:0/1261158412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 msgr2=0x7fd5a019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 0x7fd5a019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 -- 172.21.15.107:0/1261158412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a019e5f0 con 0x7fd5a0100680 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd59ffff640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 0x7fd5a0196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a548b640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 0x7fd5a019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a4c8a640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd594031610 tx=0x7fd5940043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59403f030 con 0x7fd5a0100680 2024-04-24T23:48:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd594003ce0 con 0x7fd5a0100680 2024-04-24T23:48:07.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a019e880 con 0x7fd5a0100680 2024-04-24T23:48:07.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.325+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594035070 con 0x7fd5a0100680 2024-04-24T23:48:07.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.326+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a019edf0 con 0x7fd5a0100680 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.327+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a0107a20 con 0x7fd5a0100680 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.328+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd594042030 con 0x7fd5a0100680 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.328+0000 7fd59dffb640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 0x7fd5900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.328+0000 7fd59ffff640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 0x7fd5900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.328+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd5940efb70 con 0x7fd5a0100680 2024-04-24T23:48:07.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.329+0000 7fd59ffff640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 0x7fd5900a5050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0197ee0 tx=0x7fd58c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.333+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd5940bc700 con 0x7fd5a0100680 2024-04-24T23:48:07.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.513+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd5a0062340 con 0x7fd5a0100680 2024-04-24T23:48:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.513+0000 7fd59dffb640 1 -- 172.21.15.107:0/1261158412 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd5940bc000 con 0x7fd5a0100680 2024-04-24T23:48:07.513 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:48:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 msgr2=0x7fd5900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 0x7fd5900a5050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0197ee0 tx=0x7fd58c008040 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 msgr2=0x7fd5a01969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd594031610 tx=0x7fd5940043b0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 shutdown_connections 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd5900a2b90 0x7fd5900a5050 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a0102230 0x7fd5a019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd5a0101870 0x7fd5a0196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 --2- 172.21.15.107:0/1261158412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a0100680 0x7fd5a01969d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 >> 172.21.15.107:0/1261158412 conn(0x7fd5a00fbbf0 msgr2=0x7fd5a0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 shutdown_connections 2024-04-24T23:48:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.515+0000 7fd5a6eee640 1 -- 172.21.15.107:0/1261158412 wait complete. 2024-04-24T23:48:07.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:07 smithi107 ceph-mon[29118]: pgmap v452: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-04-24T23:48:07.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3390343234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:07.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1376128425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:07 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1727023170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:48:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:07 smithi195 ceph-mon[33019]: pgmap v452: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-04-24T23:48:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3390343234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1376128425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:07 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1727023170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:07 smithi170 ceph-mon[32714]: pgmap v452: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-04-24T23:48:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3390343234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1376128425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:07 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1727023170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 -- 172.21.15.107:0/757042111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 msgr2=0x7fd9f0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/757042111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0100ac0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009900 tx=0x7fd9e002a410 comp rx=0 tx=0).stop 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 -- 172.21.15.107:0/757042111 shutdown_connections 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/757042111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0102250 0x7fd9f0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/757042111 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0101890 0x7fd9f0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/757042111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0100ac0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 -- 172.21.15.107:0/757042111 >> 172.21.15.107:0/757042111 conn(0x7fd9f00fbc30 msgr2=0x7fd9f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 -- 172.21.15.107:0/757042111 shutdown_connections 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.627+0000 7fd9f5d14640 1 -- 172.21.15.107:0/757042111 wait complete. 2024-04-24T23:48:07.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 Processor -- start 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 -- start start 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 0x7fd9f019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 0x7fd9f01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c320 con 0x7fd9f01006a0 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c1a0 con 0x7fd9f0101890 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9f5d14640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c4a0 con 0x7fd9f0102250 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37508/0 (socket says 172.21.15.107:37508) 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9ef7fe640 1 -- 172.21.15.107:0/1473501205 learned_addr learned my addr 172.21.15.107:0/1473501205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9eeffd640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 0x7fd9f019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9effff640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 0x7fd9f01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.628+0000 7fd9ef7fe640 1 -- 172.21.15.107:0/1473501205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 msgr2=0x7fd9f01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ef7fe640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 0x7fd9f01a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ef7fe640 1 -- 172.21.15.107:0/1473501205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 msgr2=0x7fd9f019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ef7fe640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 0x7fd9f019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ef7fe640 1 -- 172.21.15.107:0/1473501205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f01a1990 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9eeffd640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 0x7fd9f019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9effff640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 0x7fd9f01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ef7fe640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0031610 tx=0x7fd9e00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e003f030 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f01a1c20 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9e0003ce0 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0035070 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.629+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f01a2160 con 0x7fd9f01006a0 2024-04-24T23:48:07.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.630+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f01a23f0 con 0x7fd9f01006a0 2024-04-24T23:48:07.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.631+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd9e0042030 con 0x7fd9f01006a0 2024-04-24T23:48:07.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.631+0000 7fd9ecff9640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 0x7fd9d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.631+0000 7fd9eeffd640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 0x7fd9d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.632+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd9e00efe00 con 0x7fd9f01006a0 2024-04-24T23:48:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.632+0000 7fd9eeffd640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 0x7fd9d80a5050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.635+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9e00bc990 con 0x7fd9f01006a0 2024-04-24T23:48:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.810+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9f01a25c0 con 0x7fd9f01006a0 2024-04-24T23:48:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.811+0000 7fd9ecff9640 1 -- 172.21.15.107:0/1473501205 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd9e00bc290 con 0x7fd9f01006a0 2024-04-24T23:48:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:48:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 msgr2=0x7fd9d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 0x7fd9d80a5050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005ea0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 msgr2=0x7fd9f0199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0031610 tx=0x7fd9e00043b0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 shutdown_connections 2024-04-24T23:48:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fd9d80a2b90 0x7fd9d80a5050 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fd9f0102250 0x7fd9f01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f0101890 0x7fd9f019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 --2- 172.21.15.107:0/1473501205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f01006a0 0x7fd9f0199dc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 >> 172.21.15.107:0/1473501205 conn(0x7fd9f00fbc30 msgr2=0x7fd9f0069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 shutdown_connections 2024-04-24T23:48:07.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.813+0000 7fd9f5d14640 1 -- 172.21.15.107:0/1473501205 wait complete. 2024-04-24T23:48:07.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-24T23:48:07.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- 172.21.15.107:0/1899802148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c101790 msgr2=0x7f806c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- 172.21.15.107:0/1899802148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c101790 0x7f806c101bb0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8064009ca0 tx=0x7f806402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- 172.21.15.107:0/1899802148 shutdown_connections 2024-04-24T23:48:07.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- 172.21.15.107:0/1899802148 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c103340 0x7f806c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- 172.21.15.107:0/1899802148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c102980 0x7f806c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- 172.21.15.107:0/1899802148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c101790 0x7f806c101bb0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- 172.21.15.107:0/1899802148 >> 172.21.15.107:0/1899802148 conn(0x7f806c0fcd40 msgr2=0x7f806c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- 172.21.15.107:0/1899802148 shutdown_connections 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- 172.21.15.107:0/1899802148 wait complete. 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 Processor -- start 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- start start 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c101790 0x7f806c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 0x7f806c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10d410 con 0x7f806c102980 2024-04-24T23:48:07.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.923+0000 7f80730dd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10d290 con 0x7f806c103340 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f80730dd640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10d590 con 0x7f806c101790 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37526/0 (socket says 172.21.15.107:37526) 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 -- 172.21.15.107:0/3988132568 learned_addr learned my addr 172.21.15.107:0/3988132568 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f8070e79640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c101790 0x7f806c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f807167a640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 0x7f806c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 -- 172.21.15.107:0/3988132568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c101790 msgr2=0x7f806c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c101790 0x7f806c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 -- 172.21.15.107:0/3988132568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 msgr2=0x7f806c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 0x7f806c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 -- 172.21.15.107:0/3988132568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c19f6b0 con 0x7f806c102980 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f807167a640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 0x7f806c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:07.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f806bfff640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f805c00d950 tx=0x7f805c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c017070 con 0x7f806c102980 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f805c009c30 con 0x7f806c102980 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c19f9a0 con 0x7f806c102980 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00f6b0 con 0x7f806c102980 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.924+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c19fef0 con 0x7f806c102980 2024-04-24T23:48:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.925+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c108ab0 con 0x7f806c102980 2024-04-24T23:48:07.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.926+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f805c0109d0 con 0x7f806c102980 2024-04-24T23:48:07.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.927+0000 7f8069ffb640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 0x7f80580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:07.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.927+0000 7f8070e79640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 0x7f80580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:07.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.927+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f805c0cf060 con 0x7f806c102980 2024-04-24T23:48:07.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.927+0000 7f8070e79640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 0x7f80580a5050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8064006010 tx=0x7f8064000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:07.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:07.931+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f805c009de0 con 0x7f806c102980 2024-04-24T23:48:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.157+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f806c062340 con 0x7f806c102980 2024-04-24T23:48:08.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.165+0000 7f8069ffb640 1 -- 172.21.15.107:0/3988132568 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+740379 (secure 0 0 0) 0x7f805c09bbf0 con 0x7f806c102980 2024-04-24T23:48:08.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:08.175 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:08.175 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 18, 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "active_gid": 14152, 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "active_name": "x", 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "active_addrs": { 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6800", 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1159279165 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-04-24T23:48:08.176 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1159279165 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "active_addr": "172.21.15.107:6801/1159279165", 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "active_change": "2024-04-24T23:35:47.891537+0000", 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "active_mgr_features": 4540701547738038271, 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "available": true, 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "standbys": [ 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 24109, 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "z", 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_features": 4540701547738038271, 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.178 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.179 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.180 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.181 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.182 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.183 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.185 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.187 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-04-24T23:48:08.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.190 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.191 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.192 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.195 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.196 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T23:48:08.197 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T23:48:08.198 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.199 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.201 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.202 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.203 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.204 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.205 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.206 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.207 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.208 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.209 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.210 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-04-24T23:48:08.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T23:48:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.215 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.216 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-04-24T23:48:08.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.218 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.221 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.224 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-04-24T23:48:08.225 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-04-24T23:48:08.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T23:48:08.228 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T23:48:08.229 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T23:48:08.230 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.231 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.232 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.233 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.234 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.235 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.236 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.238 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-04-24T23:48:08.240 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T23:48:08.241 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.242 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.243 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.244 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.246 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-04-24T23:48:08.247 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-04-24T23:48:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.249 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.250 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.251 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.252 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.253 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.254 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.255 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.256 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.257 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-04-24T23:48:08.258 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.259 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.260 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.262 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-04-24T23:48:08.263 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.264 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-04-24T23:48:08.265 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-04-24T23:48:08.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.267 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.269 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.270 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.271 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.272 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.273 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.274 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.275 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.278 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.279 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.280 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.282 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.284 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-04-24T23:48:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.288 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.289 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.290 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.291 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.292 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.293 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.294 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.295 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.296 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.298 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.300 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.302 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.303 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.304 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.305 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.306 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.307 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.308 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.309 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.310 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.311 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.312 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.314 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.315 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T23:48:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T23:48:08.317 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T23:48:08.318 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T23:48:08.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T23:48:08.320 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.321 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.322 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T23:48:08.323 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.326 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.327 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-04-24T23:48:08.328 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.329 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.330 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.331 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.334 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.335 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-04-24T23:48:08.336 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.337 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.338 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-04-24T23:48:08.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.341 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.342 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.343 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.344 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.347 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.349 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.351 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.353 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.354 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.355 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.356 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.357 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.358 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.359 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.360 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.362 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.363 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.364 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.366 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.367 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-04-24T23:48:08.368 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.369 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-04-24T23:48:08.370 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.371 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.372 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.373 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.374 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.375 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.376 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-04-24T23:48:08.377 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:08.378 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.380 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:08.381 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.384 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.385 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.386 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.387 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.388 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.389 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.390 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.393 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-04-24T23:48:08.394 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.395 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.396 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.398 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.399 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.401 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.402 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.404 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.406 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-04-24T23:48:08.407 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-04-24T23:48:08.408 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.409 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.410 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.411 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.412 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.413 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.414 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.415 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.417 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.418 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.420 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.422 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.424 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.425 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.426 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.427 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.428 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.429 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.431 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.432 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.433 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.434 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.437 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.438 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.439 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.440 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.441 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.442 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.443 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.445 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.446 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.449 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.451 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.453 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-04-24T23:48:08.454 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.455 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.460 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.461 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.462 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.463 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.466 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.468 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.469 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.471 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.472 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.473 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.476 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.477 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.478 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.479 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.480 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.481 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.482 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.483 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.484 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-04-24T23:48:08.485 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-04-24T23:48:08.486 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-04-24T23:48:08.487 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.489 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.490 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.491 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.493 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.494 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.495 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.496 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.497 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-04-24T23:48:08.498 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.499 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.501 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.503 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.504 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.505 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.507 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.508 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-04-24T23:48:08.510 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.511 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.512 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.513 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.514 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-04-24T23:48:08.516 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-04-24T23:48:08.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-04-24T23:48:08.518 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-04-24T23:48:08.519 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.520 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.521 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-04-24T23:48:08.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-04-24T23:48:08.523 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-04-24T23:48:08.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.525 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-04-24T23:48:08.526 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T23:48:08.528 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-04-24T23:48:08.529 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.530 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.531 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.533 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.534 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.535 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.536 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.537 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.538 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.541 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.542 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.543 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.544 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.545 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.546 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.547 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.548 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.549 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.550 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.551 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.552 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.553 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.554 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.555 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T23:48:08.556 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T23:48:08.557 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.558 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.559 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.560 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.561 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.562 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.566 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.567 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.568 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.569 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.570 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-04-24T23:48:08.571 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-04-24T23:48:08.572 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-04-24T23:48:08.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.575 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.576 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.577 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.578 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.579 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.580 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.581 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.582 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.583 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.584 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-04-24T23:48:08.585 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.586 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.587 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.588 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.589 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.590 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.591 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.592 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.593 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.594 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.595 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.596 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.597 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.598 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.599 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:08.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 24110, 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "y", 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_features": 4540701547738038271, 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.601 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.602 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.603 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.604 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.605 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.606 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.607 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.609 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.610 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.612 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.613 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.615 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.616 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.619 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-04-24T23:48:08.620 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-04-24T23:48:08.621 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-04-24T23:48:08.622 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.623 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.624 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.626 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.627 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.628 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.628 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.628 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.628 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-04-24T23:48:08.629 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:08.630 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-04-24T23:48:08.631 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T23:48:08.632 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.633 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T23:48:08.634 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T23:48:08.635 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.636 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.637 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.638 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.639 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.640 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-04-24T23:48:08.641 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.642 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-04-24T23:48:08.643 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-04-24T23:48:08.644 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.645 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-04-24T23:48:08.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.647 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.648 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.650 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.651 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.652 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.653 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.654 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.656 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T23:48:08.657 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T23:48:08.658 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.659 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.660 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T23:48:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.662 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.663 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.664 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.665 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.666 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.667 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.668 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.669 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.670 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T23:48:08.671 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.672 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.673 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.674 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.675 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.676 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.677 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.678 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T23:48:08.679 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T23:48:08.680 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.681 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.682 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.683 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.684 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.685 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.686 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T23:48:08.687 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.688 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.689 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.690 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.692 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-04-24T23:48:08.693 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-04-24T23:48:08.694 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-04-24T23:48:08.695 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-04-24T23:48:08.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-04-24T23:48:08.697 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.698 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.699 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.700 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.703 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T23:48:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.705 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.706 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.708 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.709 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.710 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.711 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.712 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.713 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.714 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.715 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.716 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.717 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.719 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.720 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.721 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.722 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.723 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T23:48:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T23:48:08.725 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T23:48:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T23:48:08.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T23:48:08.728 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T23:48:08.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T23:48:08.730 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.731 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.732 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.733 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.734 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.735 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.736 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.737 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.738 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.740 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T23:48:08.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T23:48:08.742 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T23:48:08.743 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T23:48:08.744 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.745 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.746 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.747 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.748 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.749 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.750 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.752 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.754 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.755 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.756 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.757 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.758 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.759 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.760 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.761 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.762 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-04-24T23:48:08.763 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.764 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-04-24T23:48:08.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.767 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.768 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.769 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.771 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.772 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.773 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.774 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.775 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.776 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.777 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.778 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.779 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.780 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.781 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.782 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.783 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.784 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.785 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.786 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.787 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.788 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.789 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-04-24T23:48:08.790 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.791 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-04-24T23:48:08.792 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.794 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.795 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.796 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.797 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.798 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.799 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.800 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-04-24T23:48:08.801 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.802 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.803 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.804 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.805 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.806 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.807 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.809 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.810 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.811 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.812 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.813 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.814 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.815 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.817 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.818 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.819 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.820 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.821 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-04-24T23:48:08.823 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.824 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.825 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.826 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.827 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.829 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.830 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.834 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.835 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.837 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.838 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.839 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.843 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.844 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.846 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.849 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.850 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.852 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.854 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.855 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.856 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.858 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.859 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.860 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.862 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.863 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.864 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.865 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.866 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.867 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.869 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.870 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.873 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.874 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.876 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-04-24T23:48:08.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.878 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.879 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.881 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.882 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.883 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-04-24T23:48:08.885 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-04-24T23:48:08.887 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.888 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.889 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.890 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.891 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.892 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.893 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.894 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.895 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.897 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.898 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.899 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.900 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:08.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.903 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.906 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.907 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T23:48:08.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-04-24T23:48:08.910 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.911 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.913 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.914 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.915 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.916 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:08.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.919 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.920 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.922 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.923 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.924 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.925 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.926 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.927 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.928 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.929 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-04-24T23:48:08.930 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.932 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.933 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.934 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.935 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-04-24T23:48:08.936 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-04-24T23:48:08.937 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-04-24T23:48:08.938 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-04-24T23:48:08.939 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-04-24T23:48:08.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-04-24T23:48:08.941 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-04-24T23:48:08.942 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-04-24T23:48:08.943 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-04-24T23:48:08.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-04-24T23:48:08.945 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-04-24T23:48:08.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-04-24T23:48:08.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.953 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.954 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.955 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.956 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.957 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.960 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.961 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-04-24T23:48:08.962 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.963 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.964 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.965 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.966 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.967 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.968 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-04-24T23:48:08.969 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.970 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.971 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.972 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.973 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.974 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.975 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.976 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-04-24T23:48:08.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-04-24T23:48:08.978 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-04-24T23:48:08.979 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-04-24T23:48:08.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-04-24T23:48:08.981 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-04-24T23:48:08.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:08.983 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:08.984 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-04-24T23:48:08.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-04-24T23:48:08.986 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.988 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.989 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:08.991 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.992 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.993 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.994 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.995 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:08.996 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:08.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:08.998 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:08.999 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.000 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.003 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.004 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.005 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.006 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.007 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T23:48:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.011 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.012 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.013 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.014 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.015 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.016 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.017 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.018 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.019 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.020 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.021 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.022 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "modules": [ 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-04-24T23:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout: "dashboard", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "iostat", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "nfs", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "restful" 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.024 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.027 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.028 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.029 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.030 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.031 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.032 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.033 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.034 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.035 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.037 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.040 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.041 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:09.042 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.043 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-04-24T23:48:09.044 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-04-24T23:48:09.045 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T23:48:09.046 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.047 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T23:48:09.048 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.049 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.050 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.051 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.052 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.053 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.054 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.055 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.056 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.057 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T23:48:09.058 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.059 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:09.060 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.061 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-04-24T23:48:09.062 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.063 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.064 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.065 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-04-24T23:48:09.066 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.068 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-04-24T23:48:09.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.070 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.071 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.073 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.074 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.075 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.076 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.077 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.078 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.079 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.080 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.081 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-04-24T23:48:09.082 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-04-24T23:48:09.084 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-04-24T23:48:09.085 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.086 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.087 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.092 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T23:48:09.094 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.095 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.096 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.097 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.098 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.099 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T23:48:09.100 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.101 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.103 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.104 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.106 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-04-24T23:48:09.107 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-04-24T23:48:09.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-04-24T23:48:09.109 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.110 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.111 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.113 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.114 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.116 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.121 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.122 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T23:48:09.123 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T23:48:09.124 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.125 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.127 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.128 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.131 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.133 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T23:48:09.134 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-04-24T23:48:09.135 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.136 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T23:48:09.137 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T23:48:09.138 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T23:48:09.139 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T23:48:09.140 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T23:48:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T23:48:09.142 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.143 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.144 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.145 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.146 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.148 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:09.149 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.150 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.151 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.152 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T23:48:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T23:48:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T23:48:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T23:48:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T23:48:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T23:48:09.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T23:48:09.169 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T23:48:09.170 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T23:48:09.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T23:48:09.172 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.173 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.174 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.175 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.176 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.177 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.178 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.179 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.180 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.181 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.182 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.183 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.184 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.186 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.187 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.188 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.189 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.190 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:09.191 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.192 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-04-24T23:48:09.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.195 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.196 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.197 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.198 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.199 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-04-24T23:48:09.200 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.201 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:09.202 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.203 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:09.204 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-04-24T23:48:09.205 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.206 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.207 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.209 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.210 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.211 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.212 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.213 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.214 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-04-24T23:48:09.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.216 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.218 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.220 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.221 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.222 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.223 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-04-24T23:48:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.227 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-04-24T23:48:09.228 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.229 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.230 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.231 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.232 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.233 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.234 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.235 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.237 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.238 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.239 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.240 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.241 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T23:48:09.242 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.243 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.244 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.245 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.246 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.247 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.248 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.249 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.250 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.251 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.252 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.253 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-04-24T23:48:09.255 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.256 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.258 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.259 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.261 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.262 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.263 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.264 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.265 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.266 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.269 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.270 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.271 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.272 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.273 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.275 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.276 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.277 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.278 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.279 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.282 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.283 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.284 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.285 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.286 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.289 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.290 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.291 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.293 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.294 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.295 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.298 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.299 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.300 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.301 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.302 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.303 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.304 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-04-24T23:48:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-04-24T23:48:09.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.309 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.310 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.311 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.312 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.314 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.315 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.316 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.317 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.318 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.319 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.320 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.322 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.323 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.326 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.327 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.328 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.329 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.330 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.331 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-04-24T23:48:09.332 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.334 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.335 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-04-24T23:48:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-04-24T23:48:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-04-24T23:48:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-04-24T23:48:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-04-24T23:48:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-04-24T23:48:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-04-24T23:48:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-04-24T23:48:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-04-24T23:48:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-04-24T23:48:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-04-24T23:48:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-04-24T23:48:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-04-24T23:48:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-04-24T23:48:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-04-24T23:48:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-04-24T23:48:09.429 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-04-24T23:48:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-04-24T23:48:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-04-24T23:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "services": { 2024-04-24T23:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "dashboard": "https://172.21.15.107:8443/" 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "always_on_modules": { 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "octopus": [ 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "pacific": [ 2024-04-24T23:48:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "quincy": [ 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-04-24T23:48:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "reef": [ 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-04-24T23:48:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "squid": [ 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-04-24T23:48:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "last_failure_osd_epoch": 3, 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "active_clients": [ 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1579426836 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "libcephsqlite", 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2857053143 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3291929000 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3569834870 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-24T23:48:09.448 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1261158412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1473501205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:08 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3988132568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1261158412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1473501205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.449 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:08 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3988132568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 msgr2=0x7f80580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 0x7f80580a5050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8064006010 tx=0x7f8064000f40 comp rx=0 tx=0).stop 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 msgr2=0x7f806c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f805c00d950 tx=0x7f805c00de10 comp rx=0 tx=0).stop 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 shutdown_connections 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f80580a2b90 0x7f80580a5050 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:09.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f806c103340 0x7f806c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c102980 0x7f806c197fd0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 --2- 172.21.15.107:0/3988132568 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f806c101790 0x7f806c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 >> 172.21.15.107:0/3988132568 conn(0x7f806c0fcd40 msgr2=0x7f806c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 shutdown_connections 2024-04-24T23:48:09.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:08.171+0000 7f80730dd640 1 -- 172.21.15.107:0/3988132568 wait complete. 2024-04-24T23:48:09.457 DEBUG:tasks.check_counter:Getting stats from a 2024-04-24T23:48:09.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-24T23:48:09.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1261158412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1473501205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:09.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:08 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3988132568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T23:48:09.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:09 smithi195 ceph-mon[33019]: pgmap v453: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-24T23:48:09.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:09 smithi170 ceph-mon[32714]: pgmap v453: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-24T23:48:09.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:09 smithi107 ceph-mon[29118]: pgmap v453: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-24T23:48:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:48:11.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:11 smithi107 ceph-mon[29118]: pgmap v454: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-24T23:48:11.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:11 smithi195 ceph-mon[33019]: pgmap v454: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-24T23:48:11.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:11 smithi170 ceph-mon[32714]: pgmap v454: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-24T23:48:11.665 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 591, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 591, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 179309, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 187115, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.055319062, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.017412387, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.015062883, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.017657240, 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:11.666 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 590, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.021705714, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000036789 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:11.667 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 285, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 281, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 512468, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 114236, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.035644835, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.011044417, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.014898775, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.010183489, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 280, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.012701000, 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000045360 2024-04-24T23:48:11.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 467536, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 56928 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 2, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 70, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 428, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 193086, 2024-04-24T23:48:11.669 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.006377093, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.004228790, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.001413020, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000023491, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 68, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.004128625, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000060715 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.670 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 192768 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1303, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.036612417, 2024-04-24T23:48:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000028098 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 0, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 0, 2024-04-24T23:48:11.672 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 0, 2024-04-24T23:48:11.673 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 1, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 0, 2024-04-24T23:48:11.674 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:11.675 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:11.676 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 0 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:11.677 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-04-24T23:48:11.678 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 3, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-04-24T23:48:11.679 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 2, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 4195200, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 4194892, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 3 2024-04-24T23:48:11.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 3, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 3, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 1, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 1, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 0, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 50252, 2024-04-24T23:48:11.681 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 215420 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.682 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.683 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.684 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.685 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-04-24T23:48:11.686 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.688 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.689 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.693 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:11.693 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.694 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.695 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:11.696 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 1, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 1, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 0, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 1, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 0, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 0, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 1307, 2024-04-24T23:48:11.697 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:11.698 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:11.699 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 189319, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 50, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:11.700 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 174, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 8464, 2024-04-24T23:48:11.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 13, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 768, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 21, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 3, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 768, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 768, 2024-04-24T23:48:11.702 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.718987906, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000936182 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 768, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 769 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 768, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 768, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 0, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 469, 2024-04-24T23:48:11.703 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 297, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 1, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 0, 2024-04-24T23:48:11.704 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-04-24T23:48:11.705 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 93, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 4, 2024-04-24T23:48:11.706 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 5, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 5, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 2, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 0 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-04-24T23:48:11.707 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:11.708 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 874, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 605167, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 874, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 874, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 605167, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 768, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 901, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 768, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 901, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.710 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 768, 2024-04-24T23:48:11.711 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 768, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 768, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 768, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:11.712 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x5608eab9c6e0": { 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-04-24T23:48:11.713 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:11.715 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:12.522 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-24T23:48:12.522 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-24T23:48:12.522 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-04-24T23:48:12.522 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-04-24T23:48:12.522 DEBUG:tasks.check_counter:Getting stats from d 2024-04-24T23:48:12.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-24T23:48:12.760 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:48:13.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:13 smithi195 ceph-mon[33019]: pgmap v455: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-24T23:48:13.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:13 smithi170 ceph-mon[32714]: pgmap v455: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-24T23:48:13.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:13 smithi107 ceph-mon[29118]: pgmap v455: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 4, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 4, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 3024, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 448, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-04-24T23:48:14.576 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001655589, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000045331, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000771383, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 3, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000854525, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000284841 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 1, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 74, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 160, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 90640, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 3, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.006229371, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.004235347, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.001206575, 2024-04-24T23:48:14.578 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 72, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.003760471, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000052228 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 90640 2024-04-24T23:48:14.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 101, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 98, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 444080, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 59232, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.016900735, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.006252145, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.010415409, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.001916337, 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:14.580 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 97, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.006350645, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000065470 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 444080, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 59232 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-04-24T23:48:14.581 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:14.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:14.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 148863, 2024-04-24T23:48:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 28, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-04-24T23:48:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 102, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 429852, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:14.586 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 102, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 102, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 429852, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:14.587 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:15.307 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-24T23:48:15.307 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-24T23:48:15.307 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-24T23:48:15.307 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-24T23:48:15.307 DEBUG:tasks.check_counter:Getting stats from g 2024-04-24T23:48:15.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-24T23:48:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:48:15.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:15 smithi195 ceph-mon[33019]: pgmap v456: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:48:15.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:15 smithi170 ceph-mon[32714]: pgmap v456: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:48:15.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:15 smithi107 ceph-mon[29118]: pgmap v456: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-24T23:48:16.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:16 smithi195 ceph-mon[33019]: pgmap v457: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:48:16.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:16 smithi170 ceph-mon[32714]: pgmap v457: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:48:16.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:16 smithi107 ceph-mon[29118]: pgmap v457: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 404, 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 400, 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 567872, 2024-04-24T23:48:17.420 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 152654, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.050585394, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.020832887, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.017146542, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.010722554, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 399, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.042223335, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000105822 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.421 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 468080, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 58208 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 304, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 372, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 82142, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 291939, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.030398513, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.010617968, 2024-04-24T23:48:17.422 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.008189737, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.009205683, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 371, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.012717194, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000034278 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 196800 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:17.423 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 4, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 6, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1629, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 1284, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001219674, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000061291, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000994116, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000080653, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 4, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.002024406, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000506101 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:17.424 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-04-24T23:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1193, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.030136317, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000025260 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-04-24T23:48:17.426 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-04-24T23:48:17.427 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 1, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 0, 2024-04-24T23:48:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:17.429 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 0 2024-04-24T23:48:17.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-04-24T23:48:17.431 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 0, 2024-04-24T23:48:17.432 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 2, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 2, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 4194892, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 4194352, 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-04-24T23:48:17.433 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 2 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 3, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 3, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 0, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 1, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 1, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 0, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 0, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 0, 2024-04-24T23:48:17.434 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 50096, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 215420 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.435 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.436 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.438 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.442 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-04-24T23:48:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.443 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.444 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.445 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.447 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.448 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:17.449 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-04-24T23:48:17.450 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:17.451 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:17.452 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 197359, 2024-04-24T23:48:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 53, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 174, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:17.454 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 8464, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 13, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 600, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 21, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 3, 2024-04-24T23:48:17.455 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 600, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 600, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 1.469975281, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.002449958 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 600, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 601 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 600, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 600, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 0, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-04-24T23:48:17.456 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 297, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 301, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 1, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-04-24T23:48:17.458 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 93, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-04-24T23:48:17.459 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 4, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 6, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 6, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 2, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 0 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.460 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:17.461 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 708, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 579441, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 708, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 708, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 579441, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 600, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 593, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 600, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 593, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 600, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 600, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 600, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 600, 2024-04-24T23:48:17.464 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:17.465 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x56519028a6e0": { 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-04-24T23:48:17.466 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:17.467 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:18.162 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-24T23:48:18.163 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-24T23:48:18.163 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-24T23:48:18.163 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-04-24T23:48:18.163 DEBUG:tasks.check_counter:Getting stats from j 2024-04-24T23:48:18.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-24T23:48:18.402 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T23:48:19.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:19 smithi107 ceph-mon[29118]: pgmap v458: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:48:19.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:19 smithi195 ceph-mon[33019]: pgmap v458: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:48:19.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:19 smithi170 ceph-mon[32714]: pgmap v458: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-24T23:48:20.140 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-24T23:48:20.140 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 104, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 101, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 445520, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 60848, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.016526275, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.005739966, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.009903368, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.001846030, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 100, 2024-04-24T23:48:20.141 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.004363878, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000043638 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 445520, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 60848 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 5, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 78, 2024-04-24T23:48:20.142 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 3184, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 91440, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 3, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.006479864, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.004357674, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.001257756, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 77, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.008524307, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000110705 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:20.143 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 91440 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 0, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 2, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 0, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 0, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.000295383, 2024-04-24T23:48:20.144 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000241426, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:20.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:20.146 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:20.147 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:20.148 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 148863, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 28, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:20.149 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 105, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 430879, 2024-04-24T23:48:20.150 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 105, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 105, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 430879, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-24T23:48:20.151 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-24T23:48:20.935 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-24T23:48:20.935 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-24T23:48:20.935 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-24T23:48:20.935 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-24T23:48:20.935 DEBUG:tasks.check_counter:Getting stats from b 2024-04-24T23:48:20.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-24T23:48:21.185 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:48:21.307 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:21 smithi170 ceph-mon[32714]: pgmap v459: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-24T23:48:21.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:21 smithi107 ceph-mon[29118]: pgmap v459: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-24T23:48:21.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:21 smithi195 ceph-mon[33019]: pgmap v459: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 6097, 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 5977, 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1527166, 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 10315524, 2024-04-24T23:48:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 11, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.495374569, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.165205219, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.111564832, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.164191427, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 5976, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.878488315, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000147002 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 745, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000232779, 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000312 2024-04-24T23:48:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 2483, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 2408, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1121762, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 1363916, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 12, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.178170034, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.050947744, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.059239700, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.061531292, 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2407, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.179934664, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000074754 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 370, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000078840, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000213 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 509664, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 70112 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 6549, 2024-04-24T23:48:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 7879, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 2190801, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 3710001, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 55, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 9, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.503054503, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.196249256, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.427551562, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.069272391, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 7877, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.920667912, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000116880 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 8296, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.002988545, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000360 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 1681072, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 2551040 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4634, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 2.679175267, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000578156 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4235, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.131376560, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000031021 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 64, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 64, 2024-04-24T23:48:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 64, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.079453627, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001241462 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 12, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 24, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 24, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 23, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 2, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 2, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 95, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 88, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 12, 2024-04-24T23:48:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 64, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 4, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 234, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 64, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 42, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 5, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 37, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 22, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 12, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 1599, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 4160, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 4144, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 5, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 25 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 8, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 20, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 5, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 5, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-04-24T23:48:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 5, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-04-24T23:48:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 4215, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 4215, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 6, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 6, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 13114374, 2024-04-24T23:48:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 0, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4195, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 180.513041734, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.043030522 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 0 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 23, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 33, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 10, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 18, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 20, 2024-04-24T23:48:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 2, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 24, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 33, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 9, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 2, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 19, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 17, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 58200, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 231804 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 80, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 64, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 16, 2024-04-24T23:48:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 7, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.008118521, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001159788 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 28, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.009709970, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000346784 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000221110, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000221110 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 8, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.017268563, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002158570 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000295658, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000295658 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.004433452, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002216726 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-04-24T23:48:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.002870942, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001435471 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.005662854, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002831427 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001565676, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000782838 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 8, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.022575860, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002821982 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.006731021, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.002243673 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 2, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 1, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 11, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 11, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 185, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 324131, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 140, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 352, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 4, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 174, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 95714, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 656, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-04-24T23:48:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 9486, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 8938325, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 37, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 9486, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 9486, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 223.095733632, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.023518420 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-04-24T23:48:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 9486, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 9536 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 9486, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 79, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 9407, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 11, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 10, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 4200, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 25, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-04-24T23:48:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 5129, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 69, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 22, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-04-24T23:48:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-04-24T23:48:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 93, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 9, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 12, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 12, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 60, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 10 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 1, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 3585, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 1, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 4615, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 5, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 15125, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 3276495, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 15125, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 15125, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 3276495, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 9486, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 8938854, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 9486, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 8938854, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 9486, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 9486, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 9486, 2024-04-24T23:48:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 9486, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 5, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 529, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 5, 2024-04-24T23:48:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 5, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 529, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x558625bea500": { 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 4215, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 8920070, 2024-04-24T23:48:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 4215, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 4195, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 8920070, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:23.269 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-04-24T23:48:23.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:23 smithi170 ceph-mon[32714]: pgmap v460: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:23.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:23 smithi195 ceph-mon[33019]: pgmap v460: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:23.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:23 smithi107 ceph-mon[29118]: pgmap v460: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:24.072 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-24T23:48:24.072 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-24T23:48:24.073 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=8 2024-04-24T23:48:24.073 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-24T23:48:24.073 DEBUG:tasks.check_counter:Getting stats from e 2024-04-24T23:48:24.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-24T23:48:24.323 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:48:25.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:25 smithi170 ceph-mon[32714]: pgmap v461: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:25.477 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:25 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:48:25.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:25 smithi195 ceph-mon[33019]: pgmap v461: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:25.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:25 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:48:25.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:25 smithi107 ceph-mon[29118]: pgmap v461: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:25.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:25 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T23:48:26.029 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-04-24T23:48:26.029 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 584, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 584, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 4372057, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 184898, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.055668686, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.020200468, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.014493613, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.015543286, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 583, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.021324834, 2024-04-24T23:48:26.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000036577 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 120, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 195, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 562496, 2024-04-24T23:48:26.031 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 238372, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.013745502, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.005031287, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.003941258, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.003406315, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 193, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.010836367, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000056146 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:26.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 200464 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 390, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 386, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 4733305, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 149995, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 3, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:26.033 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.045553204, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.012650673, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.020615519, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.011767827, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 385, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.013684926, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000035545 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.034 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 470768, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 62544 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1619, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.039535095, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000024419 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.035 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 20, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 20, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 20, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-04-24T23:48:26.037 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:26.038 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 4, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 234, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:26.039 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:26.040 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:26.041 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 4215, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-04-24T23:48:26.042 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 6, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 13114374, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 13113775, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 4215 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 15, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 20, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 5, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 10, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 12, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 2, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 20, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 24, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 4, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-04-24T23:48:26.044 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 53592, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.045 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.046 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-04-24T23:48:26.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.048 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.049 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.050 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 1, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 2, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 1, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-04-24T23:48:26.056 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 1332, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:26.057 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:26.058 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 189239, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 48, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-04-24T23:48:26.059 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 174, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 60362, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 285, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:26.060 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 974, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 26, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 4, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 974, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 974, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.554986900, 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000569801 2024-04-24T23:48:26.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 974, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 975 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 974, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 974, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 612, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 359, 2024-04-24T23:48:26.062 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 2, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-04-24T23:48:26.063 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-04-24T23:48:26.064 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 93, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 4, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 5, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 5, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.065 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:26.066 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 1090, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 9560879, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 1090, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 1090, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 9560879, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.067 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 974, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 8920080, 2024-04-24T23:48:26.068 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 974, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 8920080, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:26.069 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 974, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 974, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 974, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 974, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:26.070 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:26.071 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x560cffbca6e0": { 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:26.072 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:26.073 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-04-24T23:48:26.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:48:26.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:48:26.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:26 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:48:26.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:48:26.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:48:26.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:26 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:48:26.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T23:48:26.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T23:48:26.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:26 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' 2024-04-24T23:48:26.964 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-24T23:48:26.965 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-24T23:48:26.965 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-24T23:48:26.965 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-24T23:48:26.965 DEBUG:tasks.check_counter:Getting stats from h 2024-04-24T23:48:26.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-24T23:48:27.208 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:48:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:27 smithi170 ceph-mon[32714]: pgmap v462: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:27.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:27 smithi195 ceph-mon[33019]: pgmap v462: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:27.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:27 smithi107 ceph-mon[29118]: pgmap v462: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 345, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 391, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 558015, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 262613, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 18, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.049702388, 2024-04-24T23:48:29.010 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.018837485, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.027102084, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.004280893, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 394, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.038446325, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000097579 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 405, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000087133, 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000215 2024-04-24T23:48:29.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 493552, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 66208 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 33, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 107, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 15251, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 248792, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 10, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.008787201, 2024-04-24T23:48:29.012 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.003638002, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.003810668, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000369902, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 105, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.018269524, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000173995 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 30, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000005694, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000189 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 201904 2024-04-24T23:48:29.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 56, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 60, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 30318, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 53325, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 12, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.006477316, 2024-04-24T23:48:29.014 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.001273538, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.005093400, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000312135, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 55, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.015189458, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000276171 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 72, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000016529, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000229 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 4960, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 2032 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 39, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001661399, 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000042599 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000007548, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000003774 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 3, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 1, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-04-24T23:48:29.017 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000339716, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000339716 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 2, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 11, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 14, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 1, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 13, 2024-04-24T23:48:29.018 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 12, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 2, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 9, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 4, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 3, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-04-24T23:48:29.019 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 5, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 10737418474, 2024-04-24T23:48:29.020 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:29.021 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 2 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 6, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 4, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 6, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:29.022 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-04-24T23:48:29.023 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 4, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-04-24T23:48:29.024 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 4, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 2, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-04-24T23:48:29.025 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4195424, 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 0, 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3, 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.005175344, 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001725114 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.026 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 0 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 17, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 17, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 16, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 17, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 1, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 14, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 14, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 0, 2024-04-24T23:48:29.027 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 1, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 1, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 54208, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 223612 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 4, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 3, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 4, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000339716, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000339716 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.029 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.030 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:29.031 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-04-24T23:48:29.032 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-04-24T23:48:29.033 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-04-24T23:48:29.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.039 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.040 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 0, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 1, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 1, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 0, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 1, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 1, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 90, 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:29.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:29.042 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:29.043 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 254711, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 83, 2024-04-24T23:48:29.044 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:29.045 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:29.045 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 174, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:29.046 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 68784, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 255, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-04-24T23:48:29.047 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 38, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 11332, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 33, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 38, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 38, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.207995077, 2024-04-24T23:48:29.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.005473554 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 38, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 60 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 38, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 38, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 10, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 0, 2024-04-24T23:48:29.049 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 3, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 6, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 6, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 20, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:29.050 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-04-24T23:48:29.051 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 93, 2024-04-24T23:48:29.052 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 4, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 8, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 8, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 14, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 0, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 1 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 1, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:29.053 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:29.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 430, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 554184, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 430, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 430, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 554184, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 38, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 11332, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 38, 2024-04-24T23:48:29.056 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 11332, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:29.057 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 38, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 38, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 38, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 38, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.058 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.059 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x561997542500": { 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 4, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1120, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 4, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 3, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1120, 2024-04-24T23:48:29.060 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:29.061 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-04-24T23:48:29.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:29 smithi170 ceph-mon[32714]: pgmap v463: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:29.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:29 smithi195 ceph-mon[33019]: pgmap v463: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:29.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:29 smithi107 ceph-mon[29118]: pgmap v463: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:29.856 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-24T23:48:29.856 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-24T23:48:29.856 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-24T23:48:29.856 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=6 2024-04-24T23:48:29.856 DEBUG:tasks.check_counter:Getting stats from k 2024-04-24T23:48:29.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-24T23:48:30.094 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T23:48:31.303 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:31 smithi170 ceph-mon[32714]: pgmap v464: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:31.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:31 smithi195 ceph-mon[33019]: pgmap v464: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:31.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:31 smithi107 ceph-mon[29118]: pgmap v464: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:31.877 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-04-24T23:48:31.877 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:31.877 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 629, 2024-04-24T23:48:31.877 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 631, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 189123, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 200029, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.059425315, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.021544363, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.014206700, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.017790420, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 629, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.024644560, 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000039180 2024-04-24T23:48:31.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 7, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 83, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 3740, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 203754, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:31.879 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 3, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.005742239, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.003323447, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.001499219, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000070861, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 82, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.005928875, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000072303 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 203056 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 119, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 115, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 471964, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 65262, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 3, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.019326923, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.006632722, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.011749645, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.002746333, 2024-04-24T23:48:31.881 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 114, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.007040671, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000061760 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 471696, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 64944 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1257, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.032741117, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000026047 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-04-24T23:48:31.883 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-04-24T23:48:31.884 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-04-24T23:48:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 1, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-04-24T23:48:31.886 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:31.887 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-04-24T23:48:31.888 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-04-24T23:48:31.889 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:31.890 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 2, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-04-24T23:48:31.891 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4194892, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 4194352, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 2 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-04-24T23:48:31.892 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 3, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 3, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 1, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 1, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 52172, 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-04-24T23:48:31.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.894 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.895 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.896 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-04-24T23:48:31.897 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.898 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.899 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:31.900 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.901 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.903 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.904 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 0, 2024-04-24T23:48:31.905 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:31.906 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:31.907 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:31.908 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 181239, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 46, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 174, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:31.909 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 8464, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 13, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 632, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 21, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 3, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 632, 2024-04-24T23:48:31.910 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 632, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.534987365, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000846498 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 632, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 633 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 632, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 632, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-04-24T23:48:31.911 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 313, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 317, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:31.912 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 1, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-04-24T23:48:31.913 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-04-24T23:48:31.914 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 93, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 4, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 4, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 4, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:31.915 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.916 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 751, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 588196, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 751, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 751, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 588196, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.917 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 632, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 593, 2024-04-24T23:48:31.918 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 632, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 593, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:31.919 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 632, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 632, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 632, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 632, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:31.920 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x560c7db166e0": { 2024-04-24T23:48:31.921 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-04-24T23:48:31.922 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.923 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-04-24T23:48:31.923 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-04-24T23:48:32.837 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-24T23:48:32.837 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-24T23:48:32.837 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-24T23:48:32.837 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-24T23:48:32.837 DEBUG:tasks.check_counter:Getting stats from c 2024-04-24T23:48:32.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-24T23:48:33.086 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:48:33.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:33 smithi107 ceph-mon[29118]: pgmap v465: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:33.460 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:33 smithi195 ceph-mon[33019]: pgmap v465: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:33.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:33 smithi170 ceph-mon[32714]: pgmap v465: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:34.870 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 180, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 174, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 498794, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 240014, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 24, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.032193204, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.008535696, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.020751663, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.002792193, 2024-04-24T23:48:34.871 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 170, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.021250143, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000125000 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 23, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000002924, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000127 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 493280, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 68736 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:34.872 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 29, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 104, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 14466, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 218766, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 8, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 5, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.008868080, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.003840956, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.004000413, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.000310697, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 102, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.016444519, 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000161220 2024-04-24T23:48:34.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 30, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000007490, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000249 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 202080 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 209, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 269, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 80382, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 117796, 2024-04-24T23:48:34.874 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 8, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.029860164, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.011166823, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.016274434, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.002238575, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 271, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.032924431, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000121492 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 437, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000118081, 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000270 2024-04-24T23:48:34.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "cct": { 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "total_workers": 1, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "unhealthy_workers": 0 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-MDSRank": { 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 40, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.001730772, 2024-04-24T23:48:34.876 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000043269 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 2, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000005837, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000002918 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "mds": { 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "request": 2, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "reply": 1, 2024-04-24T23:48:34.877 INFO:teuthology.orchestra.run.smithi195.stdout: "reply_latency": { 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 1, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000267687, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000267687 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_reply": 0, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "forward": 1, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_commit": 11, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_split": 0, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_merge": 0, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes": 14, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_top": 1, 2024-04-24T23:48:34.878 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_bottom": 13, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pinned": 12, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_expired": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_with_caps": 1, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "caps": 1, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "subtrees": 2, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse": 5, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_hit": 2, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_forward": 1, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_discover": 2, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_lock": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "load_cent": 0, 2024-04-24T23:48:34.879 INFO:teuthology.orchestra.run.smithi195.stdout: "q": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "exported": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "exported_inodes": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "imported": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "imported_inodes": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rfiles": 5, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rbytes": 10737418474, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rsnaps": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:34.880 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:34.881 INFO:teuthology.orchestra.run.smithi195.stdout: "process_request_cap_release": 0 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_cache": { 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update_receipt": 6, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_try_discover": 4, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_send_discover": 5, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_created": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_enqueued": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:34.882 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_migrated": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_started": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_completed": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_flush": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_inodestats": 0 2024-04-24T23:48:34.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_log": { 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evlrg": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evadd": 3, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evex": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evtrm": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "ev": 3, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evexg": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "evexd": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "segadd": 2, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "segex": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "segtrm": 0, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "seg": 2, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "segmjr": 2, 2024-04-24T23:48:34.884 INFO:teuthology.orchestra.run.smithi195.stdout: "segexg": 0, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "segexd": 0, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "expos": 4194304, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "wrpos": 4195200, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "rdpos": 0, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "jlat": { 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 2, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.003081222, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.001540611 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "replayed": 0 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_mem": { 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "ino": 17, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "ino+": 17, 2024-04-24T23:48:34.885 INFO:teuthology.orchestra.run.smithi195.stdout: "ino-": 0, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dir": 16, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dir+": 16, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dir-": 0, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dn": 14, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dn+": 14, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dn-": 0, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "cap": 1, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "cap+": 1, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "cap-": 0, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "rss": 51472, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "heap": 223612 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_server": { 2024-04-24T23:48:34.886 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_client_request": 2, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_request": 2, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_session": 3, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_peer_request": 0, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "req_create_latency": { 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getattr_latency": { 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.887 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "req_link_latency": { 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookup_latency": { 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.888 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupino_latency": { 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 1, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000267687, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000267687 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupname_latency": { 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lssnap_latency": { 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.891 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mkdir_latency": { 2024-04-24T23:48:34.891 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.891 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.891 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mknod_latency": { 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mksnap_latency": { 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "req_open_latency": { 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.892 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "req_readdir_latency": { 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rename_latency": { 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.893 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmdir_latency": { 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:34.894 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setattr_latency": { 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:34.895 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setlayout_latency": { 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.896 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setxattr_latency": { 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "req_symlink_latency": { 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "req_unlink_latency": { 2024-04-24T23:48:34.897 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_sessions": { 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "session_count": 1, 2024-04-24T23:48:34.898 INFO:teuthology.orchestra.run.smithi195.stdout: "session_add": 1, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "session_remove": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_open": 1, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_stale": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "total_load": 1, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "average_load": 1, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "avg_session_uptime": 196, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "mempool": { 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:34.899 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:34.900 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_items": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:34.901 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_bytes": 238225, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_items": 76, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_bytes": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_items": 174, 2024-04-24T23:48:34.902 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_items": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_bytes": 69144, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_items": 256, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_items": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_items": 0 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "objecter": { 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "op_active": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "op_laggy": 0, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send": 37, 2024-04-24T23:48:34.903 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send_bytes": 11108, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op_resend": 28, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op_reply": 37, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op_latency": { 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 37, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.194996131, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.005270165 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op_inflight": 0, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "oplen_avg": { 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 37, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 59 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op": 37, 2024-04-24T23:48:34.904 INFO:teuthology.orchestra.run.smithi195.stdout: "op_r": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "op_w": 37, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "op_rmw": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "op_pg": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_stat": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_create": 10, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_read": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_write": 2, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writefull": 6, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writesame": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_append": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_zero": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_truncate": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_delete": 6, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_mapext": 0, 2024-04-24T23:48:34.905 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_getxattr": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_setxattr": 20, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_call": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_watch": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_notify": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_other": 0, 2024-04-24T23:48:34.906 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_active": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_send": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_resend": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_ping": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_active": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_send": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_resend": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_active": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_send": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_resend": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_active": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_send": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_resend": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "command_active": 0, 2024-04-24T23:48:34.907 INFO:teuthology.orchestra.run.smithi195.stdout: "command_send": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "command_resend": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "map_epoch": 93, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "map_full": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "map_inc": 4, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_sessions": 8, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_open": 8, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_close": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_laggy": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_wr": 14, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_rd": 0, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_del": 1 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "oft": { 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_objs": 1, 2024-04-24T23:48:34.908 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_updates": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_removes": 0 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "osd-slow-ops": { 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_ops_count": 0 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "purge_queue": { 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed": 0, 2024-04-24T23:48:34.909 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 414, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 545948, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 414, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 414, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 545948, 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:34.910 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 37, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 11108, 2024-04-24T23:48:34.911 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 37, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 11108, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 1024, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:34.912 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 37, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 37, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 37, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 37, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:34.913 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.914 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle-0x56232a9b4500": { 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 3, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 896, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 3, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:34.915 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 2, 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 896, 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:34.916 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-24T23:48:35.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:35 smithi195 ceph-mon[33019]: pgmap v466: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:35.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:35 smithi170 ceph-mon[32714]: pgmap v466: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:35.612 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-24T23:48:35.612 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-24T23:48:35.612 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-24T23:48:35.612 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=6 2024-04-24T23:48:35.613 DEBUG:tasks.check_counter:Getting stats from f 2024-04-24T23:48:35.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-24T23:48:35.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:35 smithi107 ceph-mon[29118]: pgmap v466: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:35.875 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:48:37.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:37 smithi195 ceph-mon[33019]: pgmap v467: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:37.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:37 smithi170 ceph-mon[32714]: pgmap v467: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:37.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:37 smithi107 ceph-mon[29118]: pgmap v467: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:37.674 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-24T23:48:37.674 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:37.674 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 41, 2024-04-24T23:48:37.674 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 47, 2024-04-24T23:48:37.674 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 15990, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 181316, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 25, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 3, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.010988560, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.000949259, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.007455838, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.000231585, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 41, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.009167605, 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000223600 2024-04-24T23:48:37.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 45, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000009008, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000200 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 180, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 318, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 62222, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 271484, 2024-04-24T23:48:37.676 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 8, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 5, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.028882895, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.012758584, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.013067600, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.002160194, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 320, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.025958986, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000081121 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 395, 2024-04-24T23:48:37.677 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000107939, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000273 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 204048 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 183, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 175, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 501736, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 87292, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 7, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.024130885, 2024-04-24T23:48:37.678 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.007759217, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.016207076, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.003044955, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 175, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.023086609, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000131923 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 25, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000017367, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000694 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.679 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 486640, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 68912 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "cct": { 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "total_workers": 1, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "unhealthy_workers": 0 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-MDSRank": { 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 41, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.001544877, 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000037679 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 2, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000005882, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000002941 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "mds": { 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "request": 0, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "reply": 0, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "reply_latency": { 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.681 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_reply": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "forward": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_commit": 11, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_split": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_merge": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes": 13, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_top": 0, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_bottom": 13, 2024-04-24T23:48:37.682 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pinned": 12, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_expired": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "caps": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "subtrees": 2, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse": 2, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_hit": 1, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_forward": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_discover": 1, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:37.683 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_lock": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "load_cent": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "q": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "exported": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "exported_inodes": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "imported": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "imported_inodes": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rfiles": 5, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rbytes": 10737418474, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rsnaps": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:37.685 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "process_request_cap_release": 0 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_cache": { 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update": 0, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update_receipt": 3, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_try_discover": 2, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_send_discover": 3, 2024-04-24T23:48:37.686 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_created": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_enqueued": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_migrated": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:37.687 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_started": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_completed": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_flush": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_inodestats": 0 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_log": { 2024-04-24T23:48:37.688 INFO:teuthology.orchestra.run.smithi195.stdout: "evlrg": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "evadd": 2, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "evex": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "evtrm": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "ev": 2, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "evexg": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "evexd": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segadd": 2, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segex": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segtrm": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "seg": 2, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segmjr": 2, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segexg": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "segexd": 0, 2024-04-24T23:48:37.689 INFO:teuthology.orchestra.run.smithi195.stdout: "expos": 4194304, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "wrpos": 4194892, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "rdpos": 0, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "jlat": { 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 1, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.001221582, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.001221582 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "replayed": 0 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_mem": { 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "ino": 16, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "ino+": 16, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "ino-": 0, 2024-04-24T23:48:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: "dir": 15, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dir+": 16, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dir-": 1, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dn": 13, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dn+": 13, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dn-": 0, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "cap": 0, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "cap+": 0, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "cap-": 0, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "rss": 51040, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "heap": 223612 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_server": { 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:37.691 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_request": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_session": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_peer_request": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "req_create_latency": { 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getattr_latency": { 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "req_link_latency": { 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookup_latency": { 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.693 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupino_latency": { 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupname_latency": { 2024-04-24T23:48:37.694 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.695 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lssnap_latency": { 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mkdir_latency": { 2024-04-24T23:48:37.696 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mknod_latency": { 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mksnap_latency": { 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.697 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "req_open_latency": { 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "req_readdir_latency": { 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rename_latency": { 2024-04-24T23:48:37.698 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.699 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmdir_latency": { 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setattr_latency": { 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setlayout_latency": { 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setxattr_latency": { 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "req_symlink_latency": { 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.702 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "req_unlink_latency": { 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.703 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_sessions": { 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "session_count": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "session_add": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "session_remove": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_open": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_stale": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "total_load": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "average_load": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "avg_session_uptime": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "mempool": { 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:37.704 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:37.705 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:37.706 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_bytes": 245767, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_items": 76, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_items": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:37.707 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_items": 174, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_items": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_bytes": 64576, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_items": 238, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_items": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:37.708 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_items": 0 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "objecter": { 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_active": 0, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_laggy": 0, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send": 35, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send_bytes": 10710, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_resend": 33, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_reply": 35, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_latency": { 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 35, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.236995293, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.006771294 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "op_inflight": 0, 2024-04-24T23:48:37.709 INFO:teuthology.orchestra.run.smithi195.stdout: "oplen_avg": { 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 35, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 57 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "op": 35, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "op_r": 0, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "op_w": 35, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "op_rmw": 0, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "op_pg": 0, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_stat": 0, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_create": 10, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_read": 0, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_write": 1, 2024-04-24T23:48:37.710 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writefull": 5, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writesame": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_append": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_zero": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_truncate": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_delete": 6, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_mapext": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_getxattr": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_setxattr": 20, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:37.711 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_call": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_watch": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_notify": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_other": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_active": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_send": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_resend": 0, 2024-04-24T23:48:37.712 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_ping": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_active": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_send": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_resend": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_active": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_send": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_resend": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_active": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_send": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_resend": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "command_active": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "command_send": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "command_resend": 0, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "map_epoch": 93, 2024-04-24T23:48:37.713 INFO:teuthology.orchestra.run.smithi195.stdout: "map_full": 0, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "map_inc": 4, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_sessions": 9, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_open": 9, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_close": 0, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_laggy": 0, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_wr": 14, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_rd": 0, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_del": 1 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "oft": { 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_objs": 1, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:37.714 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_updates": 0, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_removes": 0 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "osd-slow-ops": { 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_ops_count": 0 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "purge_queue": { 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing": 0, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:37.715 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed": 0, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 400, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 533539, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 400, 2024-04-24T23:48:37.716 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 400, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 533539, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:37.717 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 35, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 10710, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 35, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 10710, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 1024, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 35, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 35, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 35, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 35, 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:37.719 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:37.720 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 0, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 0, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle-0x55c391fde500": { 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 2, 2024-04-24T23:48:37.721 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 588, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 2, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 1, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 588, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:37.722 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-24T23:48:38.580 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-24T23:48:38.580 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-24T23:48:38.580 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-24T23:48:38.580 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=3 2024-04-24T23:48:38.580 DEBUG:tasks.check_counter:Getting stats from i 2024-04-24T23:48:38.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-24T23:48:38.817 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:48:39.232 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:39 smithi195 ceph-mon[33019]: pgmap v468: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:39.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:39 smithi170 ceph-mon[32714]: pgmap v468: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:39.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:39 smithi107 ceph-mon[29118]: pgmap v468: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-24T23:48:40.585 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-24T23:48:40.585 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:40.585 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 55, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 59, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 27703, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 78756, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 15, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.010187721, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.001507252, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.006103384, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.000400372, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 55, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.022913325, 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000416605 2024-04-24T23:48:40.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 66, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000013314, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000201 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 201, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 342, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 72651, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 276091, 2024-04-24T23:48:40.587 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 8, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.031693388, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.013432381, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.014980890, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.002320270, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 346, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.032213650, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000093103 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 436, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000126468, 2024-04-24T23:48:40.588 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000290 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 205312 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 175, 2024-04-24T23:48:40.589 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 165, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 505672, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 194556, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 17, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 6, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.026712607, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.007070200, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.019707152, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.003078678, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 161, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.016977387, 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000105449 2024-04-24T23:48:40.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 10, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000001061, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000106 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 502000, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 68272 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "cct": { 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "total_workers": 1, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "unhealthy_workers": 0 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-MDSRank": { 2024-04-24T23:48:40.591 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 32, 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.002396173, 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000074880 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:40.592 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 2, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000008003, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000004001 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "mds": { 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "request": 0, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "reply": 0, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "reply_latency": { 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_reply": 0, 2024-04-24T23:48:40.593 INFO:teuthology.orchestra.run.smithi195.stdout: "forward": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_commit": 11, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_split": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_merge": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes": 14, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_top": 1, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_bottom": 13, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pinned": 12, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_expired": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:40.594 INFO:teuthology.orchestra.run.smithi195.stdout: "caps": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "subtrees": 2, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse": 5, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_hit": 3, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_forward": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_discover": 2, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_lock": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "load_cent": 1000, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "q": 1, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "exported": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "exported_inodes": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "imported": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "imported_inodes": 0, 2024-04-24T23:48:40.595 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rfiles": 5, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rbytes": 10737418474, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rsnaps": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:40.596 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "process_request_cap_release": 0 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_cache": { 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update": 0, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update_receipt": 8, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_try_discover": 5, 2024-04-24T23:48:40.597 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_send_discover": 6, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_created": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_enqueued": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_migrated": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_started": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_completed": 0, 2024-04-24T23:48:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_flush": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_inodestats": 0 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_log": { 2024-04-24T23:48:40.599 INFO:teuthology.orchestra.run.smithi195.stdout: "evlrg": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "evadd": 2, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "evex": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "evtrm": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "ev": 2, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "evexg": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "evexd": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segadd": 2, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segex": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segtrm": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "seg": 2, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segmjr": 2, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segexg": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "segexd": 0, 2024-04-24T23:48:40.600 INFO:teuthology.orchestra.run.smithi195.stdout: "expos": 4194304, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "wrpos": 4194892, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "rdpos": 0, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "jlat": { 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 1, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.001511003, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.001511003 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "replayed": 0 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_mem": { 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "ino": 17, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "ino+": 17, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "ino-": 0, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "dir": 16, 2024-04-24T23:48:40.601 INFO:teuthology.orchestra.run.smithi195.stdout: "dir+": 17, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dir-": 1, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dn": 14, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dn+": 14, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dn-": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "cap": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "cap+": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "cap-": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "rss": 54460, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "heap": 223612 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_server": { 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:40.602 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_request": 0, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_session": 0, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_peer_request": 0, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "req_create_latency": { 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getattr_latency": { 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:40.603 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "req_link_latency": { 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookup_latency": { 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.604 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupino_latency": { 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupname_latency": { 2024-04-24T23:48:40.605 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lssnap_latency": { 2024-04-24T23:48:40.606 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.607 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.607 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.607 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mkdir_latency": { 2024-04-24T23:48:40.607 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mknod_latency": { 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mksnap_latency": { 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.608 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "req_open_latency": { 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "req_readdir_latency": { 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rename_latency": { 2024-04-24T23:48:40.609 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmdir_latency": { 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setattr_latency": { 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setlayout_latency": { 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setxattr_latency": { 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "req_symlink_latency": { 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "req_unlink_latency": { 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.614 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_sessions": { 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "session_count": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "session_add": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "session_remove": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_open": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_stale": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "total_load": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "average_load": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "avg_session_uptime": 0, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "mempool": { 2024-04-24T23:48:40.615 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:40.616 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:40.617 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_bytes": 229639, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_items": 68, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_items": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:40.618 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_items": 174, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_items": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_bytes": 68760, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_items": 252, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_items": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_items": 0 2024-04-24T23:48:40.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "objecter": { 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_active": 0, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_laggy": 0, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send": 35, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send_bytes": 10710, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_resend": 33, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_reply": 35, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_latency": { 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 35, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.255994925, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.007314140 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "op_inflight": 0, 2024-04-24T23:48:40.620 INFO:teuthology.orchestra.run.smithi195.stdout: "oplen_avg": { 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 35, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 57 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "op": 35, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "op_r": 0, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "op_w": 35, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "op_rmw": 0, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "op_pg": 0, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_stat": 0, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_create": 10, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_read": 0, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_write": 1, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writefull": 5, 2024-04-24T23:48:40.621 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writesame": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_append": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_zero": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_truncate": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_delete": 6, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_mapext": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_getxattr": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_setxattr": 20, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_call": 0, 2024-04-24T23:48:40.622 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_watch": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_notify": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_other": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_active": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_send": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_resend": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_ping": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_active": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_send": 0, 2024-04-24T23:48:40.623 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_resend": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_active": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_send": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_resend": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_active": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_send": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_resend": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "command_active": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "command_send": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "command_resend": 0, 2024-04-24T23:48:40.624 INFO:teuthology.orchestra.run.smithi195.stdout: "map_epoch": 93, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "map_full": 0, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "map_inc": 4, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_sessions": 9, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_open": 9, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_close": 0, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_laggy": 0, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_wr": 14, 2024-04-24T23:48:40.625 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_rd": 0, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_del": 1 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "oft": { 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_objs": 1, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_updates": 0, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_removes": 0 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "osd-slow-ops": { 2024-04-24T23:48:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_ops_count": 0 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "purge_queue": { 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:40.627 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 427, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 556967, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 427, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 427, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 556967, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 35, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 10710, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 35, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 10710, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.629 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 1024, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 35, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 35, 2024-04-24T23:48:40.630 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 35, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 35, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:40.631 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle-0x55d643d4e500": { 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:40.632 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 2, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 588, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 2, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 1, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 588, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.633 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:40.634 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-24T23:48:41.272 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:41 smithi195 ceph-mon[33019]: pgmap v469: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:41.308 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-24T23:48:41.308 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-24T23:48:41.308 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-24T23:48:41.308 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=8 2024-04-24T23:48:41.309 DEBUG:tasks.check_counter:Getting stats from l 2024-04-24T23:48:41.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-24T23:48:41.548 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T23:48:41.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:41 smithi170 ceph-mon[32714]: pgmap v469: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:41.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:41 smithi107 ceph-mon[29118]: pgmap v469: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-24T23:48:43.295 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-04-24T23:48:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 122, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 118, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 473346, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 67188, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 1, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.021203246, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.007148466, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.012687368, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.002780412, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 117, 2024-04-24T23:48:43.296 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.006545684, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000055946 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 472512, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 66864 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 239, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 318, 2024-04-24T23:48:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 60770, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 281233, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 5, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 4, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.029742716, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.009762948, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.007185314, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.009763860, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 317, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.017748564, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000055989 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 206816 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_messages": 646, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages": 648, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_bytes": 194161, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_bytes": 205486, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_created_connections": 4, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_active_connections": 3, 2024-04-24T23:48:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_total_time": 0.066023580, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_send_time": 0.020823156, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_recv_time": 0.017315430, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_running_fast_dispatch_time": 0.021777987, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 646, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.031689513, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000049054 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_handle_ack_lat": { 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "cct": { 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "total_workers": 1, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "unhealthy_workers": 0 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-MDSRank": { 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 1494, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.046880185, 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000031378 2024-04-24T23:48:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "finisher-PurgeQueue": { 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "queue_len": 0, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "complete_latency": { 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "mds": { 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "request": 0, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "reply": 0, 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "reply_latency": { 2024-04-24T23:48:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_reply": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "forward": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_complete": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_fetch_keys": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_commit": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_split": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_merge": 0, 2024-04-24T23:48:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_top": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_bottom": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pin_tail": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_pinned": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_expired": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "inodes_with_caps": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "caps": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "subtrees": 1, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_hit": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_forward": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_discover": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_dir_fetch": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_remote_ino": 0, 2024-04-24T23:48:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "traverse_lock": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "load_cent": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "q": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "exported": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "exported_inodes": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "imported": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "imported_inodes": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_dir_fetch": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_backtrace_fetch": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "openino_peer_discover": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rfiles": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rbytes": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "root_rsnaps": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_set_tag": 0, 2024-04-24T23:48:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_inotable_repaired": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_inodes": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "scrub_file_inodes": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_inode_file_caps": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_grant": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps": 0, 2024-04-24T23:48:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_caps_dirty": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_cap_release": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "process_request_cap_release": 0 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_cache": { 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_update_receipt": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_try_discover": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_send_discover": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "dir_handle_discover": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_delayed": 0, 2024-04-24T23:48:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "num_strays_enqueuing": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_created": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_enqueued": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_reintegrated": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "strays_migrated": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_processing": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_enqueued": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "num_recovering_prioritized": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_started": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "recovery_completed": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_path": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_quiesce_inode": 0, 2024-04-24T23:48:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_exportdir": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_flush": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragmentdir": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_fragstats": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ireq_inodestats": 0 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_log": { 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evlrg": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evadd": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evex": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evtrm": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "ev": 4, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evexg": 0, 2024-04-24T23:48:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "evexd": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segadd": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segex": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segtrm": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "seg": 2, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segmjr": 2, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segexg": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "segexd": 0, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "expos": 4194304, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "wrpos": 4195424, 2024-04-24T23:48:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "rdpos": 4195283, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "jlat": { 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "replayed": 4 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_mem": { 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "ino": 3, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "ino+": 3, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "ino-": 0, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "dir": 1, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "dir+": 1, 2024-04-24T23:48:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "dir-": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "dn": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "dn+": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "dn-": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "cap": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "cap+": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "cap-": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "rss": 49904, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "heap": 215420 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_server": { 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_client_request": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "dispatch_server_request": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_request": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_client_session": 0, 2024-04-24T23:48:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "handle_peer_request": 0, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "req_create_latency": { 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getattr_latency": { 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getfilelock_latency": { 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "req_link_latency": { 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookup_latency": { 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookuphash_latency": { 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupino_latency": { 2024-04-24T23:48:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupname_latency": { 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupparent_latency": { 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lookupsnap_latency": { 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "req_lssnap_latency": { 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mkdir_latency": { 2024-04-24T23:48:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mknod_latency": { 2024-04-24T23:48:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "req_mksnap_latency": { 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "req_open_latency": { 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "req_readdir_latency": { 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rename_latency": { 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "req_renamesnap_latency": { 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "req_snapdiff_latency": { 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmdir_latency": { 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmsnap_latency": { 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "req_rmxattr_latency": { 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setattr_latency": { 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setdirlayout_latency": { 2024-04-24T23:48:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setfilelock_latency": { 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setlayout_latency": { 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "req_setxattr_latency": { 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "req_symlink_latency": { 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "req_unlink_latency": { 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_revoke_eviction": 0, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "cap_acquisition_throttle": 0, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "req_getvxattr_latency": { 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_sessions": { 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "session_count": 0, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "session_add": 1, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "session_remove": 1, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_open": 1, 2024-04-24T23:48:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "sessions_stale": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "total_load": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "average_load": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "avg_session_uptime": 1118, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "mdthresh_evicted": 0 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "mempool": { 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_bytes": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bloom_filter_items": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_alloc_items": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_data_items": 0, 2024-04-24T23:48:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_other_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_extent_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_blob_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T23:48:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_fsck_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_txc_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluestore_writing_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_reader_items": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T23:48:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "bluefs_file_writer_items": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_bytes": 189256, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_anon_items": 49, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_bytes": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "buffer_meta_items": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_bytes": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_items": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_bytes": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_mapbl_items": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_bytes": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_pglog_items": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_bytes": 9080, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_items": 174, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T23:48:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "osdmap_mapping_items": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_bytes": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "pgmap_items": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_bytes": 8464, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "mds_co_items": 13, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_bytes": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_1_items": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_bytes": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "unittest_2_items": 0 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "objecter": { 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "op_active": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "op_laggy": 0, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send": 881, 2024-04-24T23:48:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "op_send_bytes": 21, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op_resend": 3, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op_reply": 881, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op_latency": { 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 881, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 3.852924639, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.004373353 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op_inflight": 0, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "oplen_avg": { 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 881, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 882 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op": 881, 2024-04-24T23:48:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "op_r": 881, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "op_w": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "op_rmw": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "op_pg": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_stat": 553, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_create": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_read": 326, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_write": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writefull": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_writesame": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_append": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_zero": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_truncate": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_delete": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_mapext": 0, 2024-04-24T23:48:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_sparse_read": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_clonerange": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_getxattr": 1, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_setxattr": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_cmpxattr": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_rmxattr": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_resetxattrs": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_call": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_watch": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_notify": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_pgls_filter": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "osdop_other": 0, 2024-04-24T23:48:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_active": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_send": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_resend": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "linger_ping": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_active": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_send": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolop_resend": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_active": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_send": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "poolstat_resend": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_active": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_send": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "statfs_resend": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "command_active": 0, 2024-04-24T23:48:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "command_send": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "command_resend": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "map_epoch": 93, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "map_full": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "map_inc": 4, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_sessions": 5, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_open": 5, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_session_close": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "osd_laggy": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_wr": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_rd": 2, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_del": 0 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "oft": { 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_objs": 0, 2024-04-24T23:48:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_kv_pairs": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_updates": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "omap_total_removes": 0 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "osd-slow-ops": { 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "slow_ops_count": 0 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "purge_queue": { 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executing_high_water": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed_ops": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_executed": 0, 2024-04-24T23:48:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "pq_item_in_journal": 0 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 1003, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 628810, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 1003, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 1003, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 628810, 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_bytes": { 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 104857600, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 881, 2024-04-24T23:48:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 1125, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 881, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 1125, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-objecter_ops": { 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 1024, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:43.339 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 881, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 881, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 881, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 881, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle": { 2024-04-24T23:48:43.340 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.341 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "throttle-write_buf_throttle-0x55ea20cb66e0": { 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "val": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "max": 3758096384, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "get_started": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "get": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "get_sum": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_fail": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "get_or_fail_success": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "take": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "take_sum": 0, 2024-04-24T23:48:43.342 INFO:teuthology.orchestra.run.smithi195.stdout: "put": 0, 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: "put_sum": 0, 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: "wait": { 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: "avgcount": 0, 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: "sum": 0.000000000, 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: "avgtime": 0.000000000 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-04-24T23:48:43.343 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-04-24T23:48:43.343 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:43 smithi195 ceph-mon[33019]: pgmap v470: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:43.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:43 smithi170 ceph-mon[32714]: pgmap v470: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:43.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:43 smithi107 ceph-mon[29118]: pgmap v470: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-04-24T23:48:44.212 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-24T23:48:44.213 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-24T23:48:44.213 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-24T23:48:44.213 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-24T23:48:44.213 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-24T23:48:44.226 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-24T23:48:44.226 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:48:44.226 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:48:44.247 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-24T23:48:44.247 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:48:44.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 -- 172.21.15.107:0/218966993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 msgr2=0x7f123c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 --2- 172.21.15.107:0/218966993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c102e20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f1224009900 tx=0x7f122402a410 comp rx=0 tx=0).stop 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 -- 172.21.15.107:0/218966993 shutdown_connections 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 --2- 172.21.15.107:0/218966993 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c103360 0x7f123c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 --2- 172.21.15.107:0/218966993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c102e20 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 --2- 172.21.15.107:0/218966993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c1017b0 0x7f123c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 -- 172.21.15.107:0/218966993 >> 172.21.15.107:0/218966993 conn(0x7f123c0fcd40 msgr2=0x7f123c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 -- 172.21.15.107:0/218966993 shutdown_connections 2024-04-24T23:48:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.458+0000 7f12415a2640 1 -- 172.21.15.107:0/218966993 wait complete. 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 Processor -- start 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 -- start start 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c1017b0 0x7f123c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 0x7f123c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10d430 con 0x7f123c1029a0 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10d2b0 con 0x7f123c103360 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f12415a2640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10d5b0 con 0x7f123c1017b0 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f1233fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f1233fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50132/0 (socket says 172.21.15.107:50132) 2024-04-24T23:48:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f1233fff640 1 -- 172.21.15.107:0/1112001351 learned_addr learned my addr 172.21.15.107:0/1112001351 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f123affd640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c1017b0 0x7f123c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.459+0000 7f123b7fe640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 0x7f123c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 -- 172.21.15.107:0/1112001351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c1017b0 msgr2=0x7f123c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c1017b0 0x7f123c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 -- 172.21.15.107:0/1112001351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 msgr2=0x7f123c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 0x7f123c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 -- 172.21.15.107:0/1112001351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c19f6a0 con 0x7f123c1029a0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f123b7fe640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 0x7f123c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1233fff640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1224009230 tx=0x7f1224004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122403f030 con 0x7f123c1029a0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224003ce0 con 0x7f123c1029a0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123c19f930 con 0x7f123c1029a0 2024-04-24T23:48:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224035070 con 0x7f123c1029a0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.460+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f123c19fe20 con 0x7f123c1029a0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.461+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f123c108ad0 con 0x7f123c1029a0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.462+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1224042030 con 0x7f123c1029a0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.463+0000 7f1238ff9640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 0x7f12280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.463+0000 7f123affd640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 0x7f12280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.463+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f12240efe30 con 0x7f123c1029a0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.463+0000 7f123affd640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 0x7f12280a5050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f122c005fd0 tx=0x7f122c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:44.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.467+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12240bc9c0 con 0x7f123c1029a0 2024-04-24T23:48:44.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.643+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f123c062340 con 0x7f123c1029a0 2024-04-24T23:48:44.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.645+0000 7f1238ff9640 1 -- 172.21.15.107:0/1112001351 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7f12240bc2c0 con 0x7f123c1029a0 2024-04-24T23:48:44.645 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:44.645 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 msgr2=0x7f12280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 0x7f12280a5050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f122c005fd0 tx=0x7f122c00b410 comp rx=0 tx=0).stop 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 msgr2=0x7f123c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1224009230 tx=0x7f1224004360 comp rx=0 tx=0).stop 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 shutdown_connections 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f12280a2b90 0x7f12280a5050 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f123c103360 0x7f123c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f123c1029a0 0x7f123c197fc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 --2- 172.21.15.107:0/1112001351 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f123c1017b0 0x7f123c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 >> 172.21.15.107:0/1112001351 conn(0x7f123c0fcd40 msgr2=0x7f123c105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:44.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 shutdown_connections 2024-04-24T23:48:44.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.647+0000 7f12415a2640 1 -- 172.21.15.107:0/1112001351 wait complete. 2024-04-24T23:48:44.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:38:10.214052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:42:48.450218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:42:49.587829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:42:53.528878+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:48:44.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:48:44.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 -- 172.21.15.107:0/4119140951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 msgr2=0x7f3df8100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/4119140951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8100ab0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009900 tx=0x7f3de802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:44.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 -- 172.21.15.107:0/4119140951 shutdown_connections 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/4119140951 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df8108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/4119140951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/4119140951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8100ab0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 -- 172.21.15.107:0/4119140951 >> 172.21.15.107:0/4119140951 conn(0x7f3df80fbc00 msgr2=0x7f3df80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 -- 172.21.15.107:0/4119140951 shutdown_connections 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.760+0000 7f3dfd89a640 1 -- 172.21.15.107:0/4119140951 wait complete. 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 Processor -- start 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- start start 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810c310 con 0x7f3df8100690 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810c190 con 0x7f3df8101880 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810c490 con 0x7f3df8102240 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50146/0 (socket says 172.21.15.107:50146) 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 -- 172.21.15.107:0/1250691594 learned_addr learned my addr 172.21.15.107:0/1250691594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3deffff640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df77fe640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 -- 172.21.15.107:0/1250691594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 msgr2=0x7f3df819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 -- 172.21.15.107:0/1250691594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 msgr2=0x7f3df8196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 -- 172.21.15.107:0/1250691594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df8069ce0 con 0x7f3df8100690 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3deffff640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df77fe640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3df6ffd640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3de8031610 tx=0x7f3de8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de803f030 con 0x7f3df8100690 2024-04-24T23:48:44.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de8003ad0 con 0x7f3df8100690 2024-04-24T23:48:44.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df8069f70 con 0x7f3df8100690 2024-04-24T23:48:44.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8035070 con 0x7f3df8100690 2024-04-24T23:48:44.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.761+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df806a460 con 0x7f3df8100690 2024-04-24T23:48:44.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.763+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df8107a50 con 0x7f3df8100690 2024-04-24T23:48:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.764+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3de8042030 con 0x7f3df8100690 2024-04-24T23:48:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.764+0000 7f3dfc898640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 0x7f3de40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.764+0000 7f3deffff640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 0x7f3de40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.764+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3de80efbf0 con 0x7f3df8100690 2024-04-24T23:48:44.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.765+0000 7f3deffff640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 0x7f3de40a5050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3df8197f20 tx=0x7f3de0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:44.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.769+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3de80bc780 con 0x7f3df8100690 2024-04-24T23:48:44.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.942+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3df806a6c0 con 0x7f3df8100690 2024-04-24T23:48:44.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.942+0000 7f3dfc898640 1 -- 172.21.15.107:0/1250691594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f3de80bc080 con 0x7f3df8100690 2024-04-24T23:48:44.942 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-04-24T23:48:44.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.944+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 msgr2=0x7f3de40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.944+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 0x7f3de40a5050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3df8197f20 tx=0x7f3de0005f90 comp rx=0 tx=0).stop 2024-04-24T23:48:44.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.944+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 msgr2=0x7f3df8196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.944+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3de8031610 tx=0x7f3de8004270 comp rx=0 tx=0).stop 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 shutdown_connections 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f3de40a2b90 0x7f3de40a5050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3df8102240 0x7f3df819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3df8101880 0x7f3df8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 --2- 172.21.15.107:0/1250691594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8100690 0x7f3df8196a10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 >> 172.21.15.107:0/1250691594 conn(0x7f3df80fbc00 msgr2=0x7f3df8104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 shutdown_connections 2024-04-24T23:48:44.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:44.945+0000 7f3dfd89a640 1 -- 172.21.15.107:0/1250691594 wait complete. 2024-04-24T23:48:44.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 -- 172.21.15.107:0/3271844648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec100700 msgr2=0x7fdeec100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3271844648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec100700 0x7fdeec100b20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009900 tx=0x7fdee802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 -- 172.21.15.107:0/3271844648 shutdown_connections 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3271844648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1022b0 0x7fdeec108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3271844648 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec1018f0 0x7fdeec101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3271844648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec100700 0x7fdeec100b20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 -- 172.21.15.107:0/3271844648 >> 172.21.15.107:0/3271844648 conn(0x7fdeec0fbc90 msgr2=0x7fdeec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 -- 172.21.15.107:0/3271844648 shutdown_connections 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.057+0000 7fdef3f90640 1 -- 172.21.15.107:0/3271844648 wait complete. 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 Processor -- start 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 -- start start 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec100700 0x7fdeec197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1018f0 0x7fdeec198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec10c310 con 0x7fdeec1022b0 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec10c190 con 0x7fdeec1018f0 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec10c490 con 0x7fdeec100700 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50168/0 (socket says 172.21.15.107:50168) 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 -- 172.21.15.107:0/3494773175 learned_addr learned my addr 172.21.15.107:0/3494773175 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef152b640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1018f0 0x7fdeec198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef1d2c640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec100700 0x7fdeec197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 -- 172.21.15.107:0/3494773175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec100700 msgr2=0x7fdeec197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec100700 0x7fdeec197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 -- 172.21.15.107:0/3494773175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1018f0 msgr2=0x7fdeec198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1018f0 0x7fdeec198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 -- 172.21.15.107:0/3494773175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec19f750 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef252d640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fdee400eb10 tx=0x7fdee400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4004b20 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.058+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeec19fa40 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.059+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdee4004cc0 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.059+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeec19ff90 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.059+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4010700 con 0x7fdeec1022b0 2024-04-24T23:48:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.059+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeec1a0220 con 0x7fdeec1022b0 2024-04-24T23:48:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.061+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdee40108a0 con 0x7fdeec1022b0 2024-04-24T23:48:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.061+0000 7fdee2ffd640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 0x7fded80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.061+0000 7fdef1d2c640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 0x7fded80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.061+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdee40d0160 con 0x7fdeec1022b0 2024-04-24T23:48:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.062+0000 7fdef1d2c640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 0x7fded80a5050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006010 tx=0x7fdee8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.065+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdee409ccf0 con 0x7fdeec1022b0 2024-04-24T23:48:45.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.243+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdeec1a03f0 con 0x7fdeec1022b0 2024-04-24T23:48:45.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.243+0000 7fdee2ffd640 1 -- 172.21.15.107:0/3494773175 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fdee409c5f0 con 0x7fdeec1022b0 2024-04-24T23:48:45.243 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 msgr2=0x7fded80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 0x7fded80a5050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006010 tx=0x7fdee8000bc0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 msgr2=0x7fdeec19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fdee400eb10 tx=0x7fdee400c2d0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 shutdown_connections 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fded80a2b90 0x7fded80a5050 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdeec1022b0 0x7fdeec19f0e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdeec1018f0 0x7fdeec198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 --2- 172.21.15.107:0/3494773175 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fdeec100700 0x7fdeec197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 >> 172.21.15.107:0/3494773175 conn(0x7fdeec0fbc90 msgr2=0x7fdeec069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 shutdown_connections 2024-04-24T23:48:45.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.245+0000 7fdef3f90640 1 -- 172.21.15.107:0/3494773175 wait complete. 2024-04-24T23:48:45.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:45 smithi107 ceph-mon[29118]: pgmap v471: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:45.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:45 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1112001351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:45.255 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:45 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1250691594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:45.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:48:45.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 -- 172.21.15.107:0/851634552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 msgr2=0x7f67f8103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 --2- 172.21.15.107:0/851634552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f8103190 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009900 tx=0x7f67e802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:45.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 -- 172.21.15.107:0/851634552 shutdown_connections 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 --2- 172.21.15.107:0/851634552 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f81001a0 0x7f67f8109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 --2- 172.21.15.107:0/851634552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f80ff7e0 0x7f67f80ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 --2- 172.21.15.107:0/851634552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f8103190 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 -- 172.21.15.107:0/851634552 >> 172.21.15.107:0/851634552 conn(0x7f67f80fb380 msgr2=0x7f67f80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 -- 172.21.15.107:0/851634552 shutdown_connections 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 -- 172.21.15.107:0/851634552 wait complete. 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.356+0000 7f67fda31640 1 Processor -- start 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 -- start start 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 0x7f67f8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 0x7f67f8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810d3f0 con 0x7f67f8102d70 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810d270 con 0x7f67f81001a0 2024-04-24T23:48:45.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67fda31640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810d570 con 0x7f67f80ff7e0 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50190/0 (socket says 172.21.15.107:50190) 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 -- 172.21.15.107:0/2793398319 learned_addr learned my addr 172.21.15.107:0/2793398319 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f67fc640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 0x7f67f8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f6ffd640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 0x7f67f8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 -- 172.21.15.107:0/2793398319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 msgr2=0x7f67f8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 0x7f67f8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 -- 172.21.15.107:0/2793398319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 msgr2=0x7f67f8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 0x7f67f8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 -- 172.21.15.107:0/2793398319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f819f6b0 con 0x7f67f8102d70 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f67fc640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 0x7f67f8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f6ffd640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 0x7f67f8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.357+0000 7f67f77fe640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f67f000cce0 tx=0x7f67f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.358+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f000f450 con 0x7f67f8102d70 2024-04-24T23:48:45.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.358+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67f000fad0 con 0x7f67f8102d70 2024-04-24T23:48:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.358+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f819f9a0 con 0x7f67f8102d70 2024-04-24T23:48:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.358+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0018c10 con 0x7f67f8102d70 2024-04-24T23:48:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.358+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f819fef0 con 0x7f67f8102d70 2024-04-24T23:48:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.359+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8108c30 con 0x7f67f8102d70 2024-04-24T23:48:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.360+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f67f000f5f0 con 0x7f67f8102d70 2024-04-24T23:48:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.360+0000 7f67fca2f640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 0x7f67e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.360+0000 7f67f6ffd640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 0x7f67e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.360+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f67f00cfb00 con 0x7f67f8102d70 2024-04-24T23:48:45.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.361+0000 7f67f6ffd640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 0x7f67e00a50d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f67e8000c00 tx=0x7f67e8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.365+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67f0014030 con 0x7f67f8102d70 2024-04-24T23:48:45.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.541+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67f8062340 con 0x7f67f8102d70 2024-04-24T23:48:45.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.542+0000 7f67fca2f640 1 -- 172.21.15.107:0/2793398319 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f67f009c690 con 0x7f67f8102d70 2024-04-24T23:48:45.541 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 msgr2=0x7f67e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 0x7f67e00a50d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f67e8000c00 tx=0x7f67e8031400 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 msgr2=0x7f67f819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f67f000cce0 tx=0x7f67f000a620 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 shutdown_connections 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f67e00a2c10 0x7f67e00a50d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67f8102d70 0x7f67f819f040 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67f81001a0 0x7f67f8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 --2- 172.21.15.107:0/2793398319 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f67f80ff7e0 0x7f67f8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 >> 172.21.15.107:0/2793398319 conn(0x7f67f80fb380 msgr2=0x7f67f8105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 shutdown_connections 2024-04-24T23:48:45.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.544+0000 7f67fda31640 1 -- 172.21.15.107:0/2793398319 wait complete. 2024-04-24T23:48:45.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:48:45.561 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:45 smithi195 ceph-mon[33019]: pgmap v471: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:45 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1112001351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:45.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:45 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1250691594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:45 smithi170 ceph-mon[32714]: pgmap v471: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-24T23:48:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:45 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1112001351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:45.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:45 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1250691594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 -- 172.21.15.107:0/1401715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2501006a0 msgr2=0x7fa250100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1401715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2501006a0 0x7fa250100ac0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa240009900 tx=0x7fa24002a410 comp rx=0 tx=0).stop 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 -- 172.21.15.107:0/1401715845 shutdown_connections 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1401715845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa250108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1401715845 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa250101890 0x7fa250101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1401715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2501006a0 0x7fa250100ac0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 -- 172.21.15.107:0/1401715845 >> 172.21.15.107:0/1401715845 conn(0x7fa2500fbc30 msgr2=0x7fa2500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 -- 172.21.15.107:0/1401715845 shutdown_connections 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 -- 172.21.15.107:0/1401715845 wait complete. 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.656+0000 7fa2556a3640 1 Processor -- start 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 -- start start 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2501006a0 0x7fa250197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa25019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c320 con 0x7fa250101890 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c1a0 con 0x7fa250102250 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c4a0 con 0x7fa2501006a0 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50198/0 (socket says 172.21.15.107:50198) 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 -- 172.21.15.107:0/1814080715 learned_addr learned my addr 172.21.15.107:0/1814080715 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24effd640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2501006a0 0x7fa250197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24f7fe640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa25019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 -- 172.21.15.107:0/1814080715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2501006a0 msgr2=0x7fa250197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2501006a0 0x7fa250197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 -- 172.21.15.107:0/1814080715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 msgr2=0x7fa25019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa25019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 -- 172.21.15.107:0/1814080715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019f710 con 0x7fa250101890 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24f7fe640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa25019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa24e7fc640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa24400d770 tx=0x7fa24400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244017070 con 0x7fa250101890 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa244005500 con 0x7fa250101890 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019fa00 con 0x7fa250101890 2024-04-24T23:48:45.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.657+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244004e00 con 0x7fa250101890 2024-04-24T23:48:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.658+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25019ff00 con 0x7fa250101890 2024-04-24T23:48:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.658+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2501a0130 con 0x7fa250101890 2024-04-24T23:48:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.659+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa244004fa0 con 0x7fa250101890 2024-04-24T23:48:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.660+0000 7fa237fff640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 0x7fa2380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.660+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa2440cee40 con 0x7fa250101890 2024-04-24T23:48:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.660+0000 7fa24effd640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 0x7fa2380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.661+0000 7fa24effd640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 0x7fa2380a5050 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa240005ab0 tx=0x7fa240000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.664+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa24409b9d0 con 0x7fa250101890 2024-04-24T23:48:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.841+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa2501a0300 con 0x7fa250101890 2024-04-24T23:48:45.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.841+0000 7fa237fff640 1 -- 172.21.15.107:0/1814080715 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa24409b2d0 con 0x7fa250101890 2024-04-24T23:48:45.841 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 msgr2=0x7fa2380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 0x7fa2380a5050 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa240005ab0 tx=0x7fa240000bc0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 msgr2=0x7fa250198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa24400d770 tx=0x7fa24400dc30 comp rx=0 tx=0).stop 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 shutdown_connections 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa2380a2b90 0x7fa2380a5050 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa250102250 0x7fa25019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa250101890 0x7fa250198080 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 --2- 172.21.15.107:0/1814080715 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa2501006a0 0x7fa250197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 >> 172.21.15.107:0/1814080715 conn(0x7fa2500fbc30 msgr2=0x7fa2500fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 shutdown_connections 2024-04-24T23:48:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.843+0000 7fa2556a3640 1 -- 172.21.15.107:0/1814080715 wait complete. 2024-04-24T23:48:45.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.953+0000 7f4bf825d640 1 -- 172.21.15.107:0/563649060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 msgr2=0x7f4bf0109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.953+0000 7f4bf825d640 1 --2- 172.21.15.107:0/563649060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf0109bd0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009900 tx=0x7f4be802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- 172.21.15.107:0/563649060 shutdown_connections 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- 172.21.15.107:0/563649060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf0109bd0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- 172.21.15.107:0/563649060 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 0x7f4bf0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- 172.21.15.107:0/563649060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- 172.21.15.107:0/563649060 >> 172.21.15.107:0/563649060 conn(0x7f4bf00fcd20 msgr2=0x7f4bf00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- 172.21.15.107:0/563649060 shutdown_connections 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- 172.21.15.107:0/563649060 wait complete. 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 Processor -- start 2024-04-24T23:48:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- start start 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 0x7f4bf0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010d3f0 con 0x7f4bf0103340 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010d270 con 0x7f4bf0101790 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf825d640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010d570 con 0x7f4bf0102980 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf67fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf67fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50210/0 (socket says 172.21.15.107:50210) 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.954+0000 7f4bf67fa640 1 -- 172.21.15.107:0/1639299740 learned_addr learned my addr 172.21.15.107:0/1639299740 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf5ff9640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf57f8640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 0x7f4bf0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 -- 172.21.15.107:0/1639299740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 msgr2=0x7f4bf0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 0x7f4bf0197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 -- 172.21.15.107:0/1639299740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 msgr2=0x7f4bf0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 -- 172.21.15.107:0/1639299740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf019f680 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf5ff9640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf67fa640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4be80314e0 tx=0x7f4be8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be803f030 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4be8003ce0 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8035070 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf019f910 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.955+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf019fe00 con 0x7f4bf0103340 2024-04-24T23:48:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.956+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf0108ab0 con 0x7f4bf0103340 2024-04-24T23:48:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.958+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4be8042030 con 0x7f4bf0103340 2024-04-24T23:48:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.958+0000 7f4be6ffd640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 0x7f4bdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.958+0000 7f4bf5ff9640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 0x7f4bdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.958+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4be80efeb0 con 0x7f4bf0103340 2024-04-24T23:48:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.959+0000 7f4bf5ff9640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 0x7f4bdc0a5050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bf01027e0 tx=0x7f4bec009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:45.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:45.962+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4be80bca40 con 0x7f4bf0103340 2024-04-24T23:48:46.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.153+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4bf002d290 con 0x7f4bf0103340 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.153+0000 7f4be6ffd640 1 -- 172.21.15.107:0/1639299740 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+601 (secure 0 0 0) 0x7f4be80bc340 con 0x7f4bf0103340 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1738433151 2024-04-25T23:35:47.891427+0000 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4054350187 2024-04-25T23:35:47.891427+0000 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4085406700 2024-04-25T23:35:47.891427+0000 2024-04-24T23:48:46.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/526392411 2024-04-25T23:35:21.924087+0000 2024-04-24T23:48:46.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1308114869 2024-04-25T23:35:21.924087+0000 2024-04-24T23:48:46.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2998673885 2024-04-25T23:35:21.924087+0000 2024-04-24T23:48:46.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/1957718163 2024-04-25T23:35:21.924087+0000 2024-04-24T23:48:46.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2063300390 2024-04-25T23:35:47.891427+0000 2024-04-24T23:48:46.154 INFO:teuthology.orchestra.run.smithi107.stderr:listed 10 entries 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 msgr2=0x7f4bdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 0x7f4bdc0a5050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4bf01027e0 tx=0x7f4bec009540 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 msgr2=0x7f4bf019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4be80314e0 tx=0x7f4be8004360 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 shutdown_connections 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f4bdc0a2b90 0x7f4bdc0a5050 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bf0103340 0x7f4bf019f010 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4bf0102980 0x7f4bf0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 --2- 172.21.15.107:0/1639299740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bf0101790 0x7f4bf0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 >> 172.21.15.107:0/1639299740 conn(0x7f4bf00fcd20 msgr2=0x7f4bf0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 shutdown_connections 2024-04-24T23:48:46.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.155+0000 7f4bf825d640 1 -- 172.21.15.107:0/1639299740 wait complete. 2024-04-24T23:48:46.167 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-24T23:48:46.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:48:46.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-24T23:48:46.204 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:48:46.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:48:46.204 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T23:48:46.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:48:46.260 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-04-24T23:48:46.316 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T23:48:46.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:48:46.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:48:46.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:48:46.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2024-04-24T23:48:46.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3494773175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2793398319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1814080715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:46 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1639299740' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:48:46.481 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-24T23:48:46.491 INFO:tasks.cephadm:Teardown begin 2024-04-24T23:48:46.492 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T23:48:46.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:48:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3494773175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2793398319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1814080715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:46 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1639299740' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:48:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3494773175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2793398319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1814080715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-24T23:48:46.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:46 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1639299740' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T23:48:46.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- 172.21.15.107:0/1817740436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c0ffe60 msgr2=0x7fcc8c100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 --2- 172.21.15.107:0/1817740436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c100280 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00aa10 tx=0x7fcc7c02b720 comp rx=0 tx=0).stop 2024-04-24T23:48:46.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- 172.21.15.107:0/1817740436 shutdown_connections 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 --2- 172.21.15.107:0/1817740436 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101a00 0x7fcc8c108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 --2- 172.21.15.107:0/1817740436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c101040 0x7fcc8c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 --2- 172.21.15.107:0/1817740436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c100280 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- 172.21.15.107:0/1817740436 >> 172.21.15.107:0/1817740436 conn(0x7fcc8c0fb420 msgr2=0x7fcc8c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- 172.21.15.107:0/1817740436 shutdown_connections 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- 172.21.15.107:0/1817740436 wait complete. 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 Processor -- start 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.622+0000 7fcc9262c640 1 -- start start 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101040 0x7fcc8c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c10b320 con 0x7fcc8c101a00 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c10b1a0 con 0x7fcc8c0ffe60 2024-04-24T23:48:46.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc9262c640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c10b4a0 con 0x7fcc8c101040 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46070/0 (socket says 172.21.15.107:46070) 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 -- 172.21.15.107:0/546893132 learned_addr learned my addr 172.21.15.107:0/546893132 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc8bfff640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc8b7fe640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101040 0x7fcc8c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 -- 172.21.15.107:0/546893132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101040 msgr2=0x7fcc8c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101040 0x7fcc8c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 -- 172.21.15.107:0/546893132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 msgr2=0x7fcc8c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 -- 172.21.15.107:0/546893132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8c1a19d0 con 0x7fcc8c101a00 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc8bfff640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc90bc9640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcc8400b970 tx=0x7fcc8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.623+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc84019070 con 0x7fcc8c101a00 2024-04-24T23:48:46.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.624+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc8c1a1cc0 con 0x7fcc8c101a00 2024-04-24T23:48:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.624+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc8400cb40 con 0x7fcc8c101a00 2024-04-24T23:48:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.624+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc84007610 con 0x7fcc8c101a00 2024-04-24T23:48:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.624+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8c1a2200 con 0x7fcc8c101a00 2024-04-24T23:48:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.625+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc8c1071a0 con 0x7fcc8c101a00 2024-04-24T23:48:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.626+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fcc8401c070 con 0x7fcc8c101a00 2024-04-24T23:48:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.626+0000 7fcc897fa640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 0x7fcc740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.627+0000 7fcc8bfff640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 0x7fcc740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.627+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc840cea30 con 0x7fcc8c101a00 2024-04-24T23:48:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.627+0000 7fcc8bfff640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 0x7fcc740a5050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c030510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.631+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc8409b5c0 con 0x7fcc8c101a00 2024-04-24T23:48:46.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.844+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc8c0623e0 con 0x7fcc8c101a00 2024-04-24T23:48:46.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.845+0000 7fcc897fa640 1 -- 172.21.15.107:0/546893132 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7fcc8409aec0 con 0x7fcc8c101a00 2024-04-24T23:48:46.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:46.845 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:48:46.845 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 msgr2=0x7fcc740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 0x7fcc740a5050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000c00 tx=0x7fcc7c030510 comp rx=0 tx=0).stop 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 msgr2=0x7fcc8c1a1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcc8400b970 tx=0x7fcc8400be30 comp rx=0 tx=0).stop 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 shutdown_connections 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fcc740a2b90 0x7fcc740a5050 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc8c101a00 0x7fcc8c1a1360 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fcc8c101040 0x7fcc8c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 --2- 172.21.15.107:0/546893132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc8c0ffe60 0x7fcc8c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 >> 172.21.15.107:0/546893132 conn(0x7fcc8c0fb420 msgr2=0x7fcc8c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:46.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 shutdown_connections 2024-04-24T23:48:46.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.847+0000 7fcc9262c640 1 -- 172.21.15.107:0/546893132 wait complete. 2024-04-24T23:48:46.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:48:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 -- 172.21.15.107:0/3671468515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624104b40 msgr2=0x7fb624106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 --2- 172.21.15.107:0/3671468515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624104b40 0x7fb624106f30 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb618009900 tx=0x7fb61802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 -- 172.21.15.107:0/3671468515 shutdown_connections 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 --2- 172.21.15.107:0/3671468515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624107470 0x7fb6241098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 --2- 172.21.15.107:0/3671468515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624104b40 0x7fb624106f30 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 --2- 172.21.15.107:0/3671468515 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624102210 0x7fb624104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 -- 172.21.15.107:0/3671468515 >> 172.21.15.107:0/3671468515 conn(0x7fb6240fbc20 msgr2=0x7fb6240fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 -- 172.21.15.107:0/3671468515 shutdown_connections 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.957+0000 7fb628c95640 1 -- 172.21.15.107:0/3671468515 wait complete. 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 Processor -- start 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 -- start start 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624104b40 0x7fb6241001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 0x7fb624100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb62410d420 con 0x7fb624102210 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb62410d2a0 con 0x7fb624104b40 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb628c95640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fb62410d5a0 con 0x7fb624107470 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb62259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb62259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46090/0 (socket says 172.21.15.107:46090) 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.958+0000 7fb62259c640 1 -- 172.21.15.107:0/3553305901 learned_addr learned my addr 172.21.15.107:0/3553305901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb622d9d640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 0x7fb624100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 -- 172.21.15.107:0/3553305901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 msgr2=0x7fb624100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 0x7fb624100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 -- 172.21.15.107:0/3553305901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624104b40 msgr2=0x7fb6241001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb621d9b640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624104b40 0x7fb6241001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624104b40 0x7fb6241001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 -- 172.21.15.107:0/3553305901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb624100f90 con 0x7fb624102210 2024-04-24T23:48:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb622d9d640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 0x7fb624100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb62259c640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00d770 tx=0x7fb61c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c017070 con 0x7fb624102210 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6241a1bf0 con 0x7fb624102210 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb61c005500 con 0x7fb624102210 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004e00 con 0x7fb624102210 2024-04-24T23:48:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.959+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6241a20f0 con 0x7fb624102210 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.960+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6240fd710 con 0x7fb624102210 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.962+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb61c004fa0 con 0x7fb624102210 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.962+0000 7fb6137fe640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 0x7fb60c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.962+0000 7fb621d9b640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 0x7fb60c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.962+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb61c0cef20 con 0x7fb624102210 2024-04-24T23:48:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.963+0000 7fb621d9b640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 0x7fb60c0a5050 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb618006010 tx=0x7fb618000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:46.967+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb61c09bab0 con 0x7fb624102210 2024-04-24T23:48:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.180+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb624062340 con 0x7fb624102210 2024-04-24T23:48:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.180+0000 7fb6137fe640 1 -- 172.21.15.107:0/3553305901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11956 (secure 0 0 0) 0x7fb61c09b3b0 con 0x7fb624102210 2024-04-24T23:48:47.180 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:47.180 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3058072142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3058072142},{"type":"v1","addr":"172.21.15.107:6835","nonce":3058072142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3538832535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3538832535},{"type":"v1","addr":"172.21.15.107:6837","nonce":3538832535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T23:42:50.405146+0000","modified":"2024-04-24T23:45:14.157584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24493,"mds_2":24464,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6839/527084969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":527084969},{"type":"v1","addr":"172.21.15.107:6839","nonce":527084969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6841/1612113576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":1612113576},{"type":"v1","addr":"172.21.15.107:6841","nonce":1612113576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24406":{"gid":24406,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.195:6833/1442081594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":1442081594},{"type":"v1","addr":"172.21.15.195:6833","nonce":1442081594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6833/3444431424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3444431424},{"type":"v1","addr":"172.21.15.170:6833","nonce":3444431424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/3179503985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3179503985},{"type":"v1","addr":"172.21.15.170:6835","nonce":3179503985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.195:6835/1997811297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6834","nonce":1997811297},{"type":"v1","addr":"172.21.15.195:6835","nonce":1997811297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.195:6837/1315131786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6836","nonce":1315131786},{"type":"v1","addr":"172.21.15.195:6837","nonce":1315131786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.170:6837/293399389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":293399389},{"type":"v1","addr":"172.21.15.170:6837","nonce":293399389}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.195:6839/4197811615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6838","nonce":4197811615},{"type":"v1","addr":"172.21.15.195:6839","nonce":4197811615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6839/2021576737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2021576737},{"type":"v1","addr":"172.21.15.170:6839","nonce":2021576737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24451,24493,24463,24464,24506]},"id":1}]} 2024-04-24T23:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-04-24T23:48:47.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 msgr2=0x7fb60c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 0x7fb60c0a5050 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb618006010 tx=0x7fb618000bc0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 msgr2=0x7fb624101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00d770 tx=0x7fb61c00dc30 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 shutdown_connections 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fb60c0a2b90 0x7fb60c0a5050 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fb624107470 0x7fb624100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb624104b40 0x7fb6241001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 --2- 172.21.15.107:0/3553305901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb624102210 0x7fb624101b80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.182+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 >> 172.21.15.107:0/3553305901 conn(0x7fb6240fbc20 msgr2=0x7fb624102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.183+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 shutdown_connections 2024-04-24T23:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.183+0000 7fb628c95640 1 -- 172.21.15.107:0/3553305901 wait complete. 2024-04-24T23:48:47.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:47.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.292+0000 7fa558658640 1 -- 172.21.15.107:0/3195480236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa550103360 msgr2=0x7fa550109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.292+0000 7fa558658640 1 --2- 172.21.15.107:0/3195480236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa550103360 0x7fa550109bf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa548009900 tx=0x7fa54802a3f0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- 172.21.15.107:0/3195480236 shutdown_connections 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- 172.21.15.107:0/3195480236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa550103360 0x7fa550109bf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- 172.21.15.107:0/3195480236 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 0x7fa550102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- 172.21.15.107:0/3195480236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5501017b0 0x7fa550101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- 172.21.15.107:0/3195480236 >> 172.21.15.107:0/3195480236 conn(0x7fa5500fcd20 msgr2=0x7fa5500ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- 172.21.15.107:0/3195480236 shutdown_connections 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- 172.21.15.107:0/3195480236 wait complete. 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 Processor -- start 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- start start 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 0x7fa550197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 0x7fa55019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d410 con 0x7fa5501017b0 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d290 con 0x7fa550103360 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa558658640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d590 con 0x7fa5501029a0 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa5563f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa5563f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46118/0 (socket says 172.21.15.107:46118) 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa5563f4640 1 -- 172.21.15.107:0/3014130 learned_addr learned my addr 172.21.15.107:0/3014130 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.293+0000 7fa555bf3640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 0x7fa550197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa556bf5640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 0x7fa55019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 -- 172.21.15.107:0/3014130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 msgr2=0x7fa550197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 0x7fa550197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 -- 172.21.15.107:0/3014130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 msgr2=0x7fa55019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 0x7fa55019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 -- 172.21.15.107:0/3014130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55019f550 con 0x7fa5501017b0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa556bf5640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 0x7fa55019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa5563f4640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ccb0 tx=0x7fa54000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540003970 con 0x7fa5501017b0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55019f840 con 0x7fa5501017b0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa540004030 con 0x7fa5501017b0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54000f6b0 con 0x7fa5501017b0 2024-04-24T23:48:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.294+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55019fd90 con 0x7fa5501017b0 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.295+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa550108a00 con 0x7fa5501017b0 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.296+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa5400041e0 con 0x7fa5501017b0 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.296+0000 7fa54f7fe640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 0x7fa5380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.297+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa5400cf5c0 con 0x7fa5501017b0 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.297+0000 7fa555bf3640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 0x7fa5380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.297+0000 7fa555bf3640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 0x7fa5380a5050 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa550198e40 tx=0x7fa54400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.301+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa540014070 con 0x7fa5501017b0 2024-04-24T23:48:47.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.484+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5500623e0 con 0x7fa5501017b0 2024-04-24T23:48:47.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.486+0000 7fa54f7fe640 1 -- 172.21.15.107:0/3014130 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24902 (secure 0 0 0) 0x7fa54009c150 con 0x7fa5501017b0 2024-04-24T23:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-24T23:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"16fa2ae8-0293-11ef-bc93-c7b262605968","created":"2024-04-24T23:34:45.996336+0000","modified":"2024-04-24T23:44:50.063628+0000","last_up_change":"2024-04-24T23:41:57.533756+0000","last_in_change":"2024-04-24T23:41:37.328805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:38:10.214052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:42:48.450218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:42:49.587829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:42:53.528878+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7adc01bb-2472-44e3-abc7-5ef350338376","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6803","nonce":1988153607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6805","nonce":1988153607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6809","nonce":1988153607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1988153607},{"type":"v1","addr":"172.21.15.107:6807","nonce":1988153607}]},"public_addr":"172.21.15.107:6803/1988153607","cluster_addr":"172.21.15.107:6805/1988153607","heartbeat_back_addr":"172.21.15.107:6809/1988153607","heartbeat_front_addr":"172.21.15.107:6807/1988153607","state":["exists","up"]},{"osd":1,"uuid":"6f9087e7-4eaa-4516-b08f-3eef8dbae2d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6801","nonce":80173128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6803","nonce":80173128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6807","nonce":80173128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":80173128},{"type":"v1","addr":"172.21.15.170:6805","nonce":80173128}]},"public_addr":"172.21.15.170:6801/80173128","cluster_addr":"172.21.15.170:6803/80173128","heartbeat_back_addr":"172.21.15.170:6807/80173128","heartbeat_front_addr":"172.21.15.170:6805/80173128","state":["exists","up"]},{"osd":2,"uuid":"010d750a-e582-46e0-ac12-95fedac8d734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6801","nonce":3967620404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6803","nonce":3967620404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6807","nonce":3967620404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3967620404},{"type":"v1","addr":"172.21.15.195:6805","nonce":3967620404}]},"public_addr":"172.21.15.195:6801/3967620404","cluster_addr":"172.21.15.195:6803/3967620404","heartbeat_back_addr":"172.21.15.195:6807/3967620404","heartbeat_front_addr":"172.21.15.195:6805/3967620404","state":["exists","up"]},{"osd":3,"uuid":"14800c24-129e-4081-bb09-cbef9af89c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6811","nonce":3900016553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6813","nonce":3900016553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6817","nonce":3900016553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":3900016553},{"type":"v1","addr":"172.21.15.107:6815","nonce":3900016553}]},"public_addr":"172.21.15.107:6811/3900016553","cluster_addr":"172.21.15.107:6813/3900016553","heartbeat_back_addr":"172.21.15.107:6817/3900016553","heartbeat_front_addr":"172.21.15.107:6815/3900016553","state":["exists","up"]},{"osd":4,"uuid":"7310213f-3a15-436a-a0bc-38c005a7aeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6809","nonce":2310043947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6811","nonce":2310043947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6815","nonce":2310043947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2310043947},{"type":"v1","addr":"172.21.15.170:6813","nonce":2310043947}]},"public_addr":"172.21.15.170:6809/2310043947","cluster_addr":"172.21.15.170:6811/2310043947","heartbeat_back_addr":"172.21.15.170:6815/2310043947","heartbeat_front_addr":"172.21.15.170:6813/2310043947","state":["exists","up"]},{"osd":5,"uuid":"e45f9071-7234-40da-87e6-04473b1a0b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6809","nonce":1022606297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6811","nonce":1022606297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6815","nonce":1022606297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":1022606297},{"type":"v1","addr":"172.21.15.195:6813","nonce":1022606297}]},"public_addr":"172.21.15.195:6809/1022606297","cluster_addr":"172.21.15.195:6811/1022606297","heartbeat_back_addr":"172.21.15.195:6815/1022606297","heartbeat_front_addr":"172.21.15.195:6813/1022606297","state":["exists","up"]},{"osd":6,"uuid":"1033c741-8643-41f0-b5e3-a7dfbb4d47d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6819","nonce":3398193587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6821","nonce":3398193587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6825","nonce":3398193587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3398193587},{"type":"v1","addr":"172.21.15.107:6823","nonce":3398193587}]},"public_addr":"172.21.15.107:6819/3398193587","cluster_addr":"172.21.15.107:6821/3398193587","heartbeat_back_addr":"172.21.15.107:6825/3398193587","heartbeat_front_addr":"172.21.15.107:6823/3398193587","state":["exists","up"]},{"osd":7,"uuid":"42a4f20d-28a4-4910-9850-ebd9c9919974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6817","nonce":2352399844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6819","nonce":2352399844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6823","nonce":2352399844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2352399844},{"type":"v1","addr":"172.21.15.170:6821","nonce":2352399844}]},"public_addr":"172.21.15.170:6817/2352399844","cluster_addr":"172.21.15.170:6819/2352399844","heartbeat_back_addr":"172.21.15.170:6823/2352399844","heartbeat_front_addr":"172.21.15.170:6821/2352399844","state":["exists","up"]},{"osd":8,"uuid":"e9844dda-7739-49ea-9548-685903258f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6817","nonce":2912271513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6819","nonce":2912271513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6823","nonce":2912271513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":2912271513},{"type":"v1","addr":"172.21.15.195:6821","nonce":2912271513}]},"public_addr":"172.21.15.195:6817/2912271513","cluster_addr":"172.21.15.195:6819/2912271513","heartbeat_back_addr":"172.21.15.195:6823/2912271513","heartbeat_front_addr":"172.21.15.195:6821/2912271513","state":["exists","up"]},{"osd":9,"uuid":"98528d16-0e91-4f6c-99ed-71e534aa6551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6827","nonce":150997445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6829","nonce":150997445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6833","nonce":150997445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":150997445},{"type":"v1","addr":"172.21.15.107:6831","nonce":150997445}]},"public_addr":"172.21.15.107:6827/150997445","cluster_addr":"172.21.15.107:6829/150997445","heartbeat_back_addr":"172.21.15.107:6833/150997445","heartbeat_front_addr":"172.21.15.107:6831/150997445","state":["exists","up"]},{"osd":10,"uuid":"eef6b39d-cb02-4439-9562-fb2ee00ed40d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6825","nonce":1507442612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6827","nonce":1507442612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6831","nonce":1507442612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":1507442612},{"type":"v1","addr":"172.21.15.170:6829","nonce":1507442612}]},"public_addr":"172.21.15.170:6825/1507442612","cluster_addr":"172.21.15.170:6827/1507442612","heartbeat_back_addr":"172.21.15.170:6831/1507442612","heartbeat_front_addr":"172.21.15.170:6829/1507442612","state":["exists","up"]},{"osd":11,"uuid":"fc99798e-5b23-45a7-9bf3-40c43d5d092e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6825","nonce":2390923874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6827","nonce":2390923874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6831","nonce":2390923874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2390923874},{"type":"v1","addr":"172.21.15.195:6829","nonce":2390923874}]},"public_addr":"172.21.15.195:6825/2390923874","cluster_addr":"172.21.15.195:6827/2390923874","heartbeat_back_addr":"172.21.15.195:6831/2390923874","heartbeat_front_addr":"172.21.15.195:6829/2390923874","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:19.738215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:37:43.260038+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:06.617925+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:31.486825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:38:56.624571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:20.967659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:39:46.228809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:12.354198+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:40:37.772385+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:03.376710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:29.160741+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:41:55.857637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1738433151":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/4054350187":"2024-04-25T23:35:47.891427+0000","172.21.15.107:6800/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/4085406700":"2024-04-25T23:35:47.891427+0000","172.21.15.107:0/526392411":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/1308114869":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2998673885":"2024-04-25T23:35:21.924087+0000","172.21.15.107:6801/1957718163":"2024-04-25T23:35:21.924087+0000","172.21.15.107:0/2063300390":"2024-04-25T23:35:47.891427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 msgr2=0x7fa5380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 0x7fa5380a5050 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa550198e40 tx=0x7fa54400a2d0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 msgr2=0x7fa550197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ccb0 tx=0x7fa54000a620 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 shutdown_connections 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fa5380a2b90 0x7fa5380a5050 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa550103360 0x7fa55019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fa5501029a0 0x7fa550197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 --2- 172.21.15.107:0/3014130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5501017b0 0x7fa550197930 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 >> 172.21.15.107:0/3014130 conn(0x7fa5500fcd20 msgr2=0x7fa550105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 shutdown_connections 2024-04-24T23:48:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.488+0000 7fa558658640 1 -- 172.21.15.107:0/3014130 wait complete. 2024-04-24T23:48:47.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-mon[29118]: pgmap v472: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:47.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/546893132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:47.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T23:48:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mon[33019]: pgmap v472: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:47.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/546893132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-mon[32714]: pgmap v472: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-24T23:48:47.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/546893132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:47.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- 172.21.15.107:0/594428668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608107460 msgr2=0x7f36081098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 --2- 172.21.15.107:0/594428668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608107460 0x7f36081098b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3600009ca0 tx=0x7f360002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- 172.21.15.107:0/594428668 shutdown_connections 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 --2- 172.21.15.107:0/594428668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608107460 0x7f36081098b0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 --2- 172.21.15.107:0/594428668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 0x7f3608106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 --2- 172.21.15.107:0/594428668 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608102200 0x7f36081045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- 172.21.15.107:0/594428668 >> 172.21.15.107:0/594428668 conn(0x7f36080fbc30 msgr2=0x7f36080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- 172.21.15.107:0/594428668 shutdown_connections 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- 172.21.15.107:0/594428668 wait complete. 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 Processor -- start 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.598+0000 7f360e4d6640 1 -- start start 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 0x7f3608197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608107460 0x7f360819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f360810d400 con 0x7f3608102200 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f360810d280 con 0x7f3608104b30 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f360810d580 con 0x7f3608107460 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46128/0 (socket says 172.21.15.107:46128) 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 -- 172.21.15.107:0/2970475080 learned_addr learned my addr 172.21.15.107:0/2970475080 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360ca73640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608107460 0x7f360819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f36077fe640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 0x7f3608197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 -- 172.21.15.107:0/2970475080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608107460 msgr2=0x7f360819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608107460 0x7f360819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 -- 172.21.15.107:0/2970475080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 msgr2=0x7f3608197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 0x7f3608197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 -- 172.21.15.107:0/2970475080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360819f560 con 0x7f3608102200 2024-04-24T23:48:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f3607fff640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c920 tx=0x7f35fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc007c70 con 0x7f3608102200 2024-04-24T23:48:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35fc007e10 con 0x7f3608102200 2024-04-24T23:48:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360819f850 con 0x7f3608102200 2024-04-24T23:48:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.599+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc018890 con 0x7f3608102200 2024-04-24T23:48:47.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.600+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f360819fda0 con 0x7f3608102200 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.600+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36080fd720 con 0x7f3608102200 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.601+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f35fc018a70 con 0x7f3608102200 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.602+0000 7f36057fa640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 0x7f35f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.602+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f35fc0cf610 con 0x7f3608102200 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.602+0000 7f36077fe640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 0x7f35f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.603+0000 7f36077fe640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 0x7f35f00a5050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005e10 tx=0x7f35f8005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.606+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35fc014030 con 0x7f3608102200 2024-04-24T23:48:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.818+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3608062370 con 0x7f3608102200 2024-04-24T23:48:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.820+0000 7f36057fa640 1 -- 172.21.15.107:0/2970475080 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) ==== 161+0+0 (secure 0 0 0) 0x7f35fc00de20 con 0x7f3608102200 2024-04-24T23:48:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 msgr2=0x7f35f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 0x7f35f00a5050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005e10 tx=0x7f35f8005da0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 msgr2=0x7f3608197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c920 tx=0x7f35fc00cde0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 shutdown_connections 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f35f00a2b90 0x7f35f00a5050 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f3608107460 0x7f360819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3608104b30 0x7f3608197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 --2- 172.21.15.107:0/2970475080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3608102200 0x7f3608197940 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 >> 172.21.15.107:0/2970475080 conn(0x7f36080fbc30 msgr2=0x7f3608102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 shutdown_connections 2024-04-24T23:48:47.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.824+0000 7f360e4d6640 1 -- 172.21.15.107:0/2970475080 wait complete. 2024-04-24T23:48:47.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T23:48:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- 172.21.15.107:0/2298688544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1074e0 msgr2=0x7f902c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 --2- 172.21.15.107:0/2298688544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1074e0 0x7f902c1098d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9024009ca0 tx=0x7f902402a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- 172.21.15.107:0/2298688544 shutdown_connections 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 --2- 172.21.15.107:0/2298688544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c1074e0 0x7f902c1098d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 --2- 172.21.15.107:0/2298688544 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c104bb0 0x7f902c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 --2- 172.21.15.107:0/2298688544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c102280 0x7f902c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- 172.21.15.107:0/2298688544 >> 172.21.15.107:0/2298688544 conn(0x7f902c0fbc30 msgr2=0x7f902c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- 172.21.15.107:0/2298688544 shutdown_connections 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- 172.21.15.107:0/2298688544 wait complete. 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 Processor -- start 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.941+0000 7f9032838640 1 -- start start 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c102280 0x7f902c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 0x7f902c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10d490 con 0x7f902c104bb0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10d310 con 0x7f902c1074e0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9032838640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10d610 con 0x7f902c102280 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46192/0 (socket says 172.21.15.107:46192) 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 -- 172.21.15.107:0/2918642015 learned_addr learned my addr 172.21.15.107:0/2918642015 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902bfff640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c102280 0x7f902c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9030dd5640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 0x7f902c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 -- 172.21.15.107:0/2918642015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c102280 msgr2=0x7f902c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c102280 0x7f902c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 -- 172.21.15.107:0/2918642015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 msgr2=0x7f902c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 0x7f902c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 -- 172.21.15.107:0/2918642015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c19f6b0 con 0x7f902c104bb0 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f9030dd5640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 0x7f902c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:47.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f902b7fe640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f902000d740 tx=0x7f902000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.942+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020017070 con 0x7f902c104bb0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.943+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c19f9a0 con 0x7f902c104bb0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.943+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f902000de50 con 0x7f902c104bb0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.943+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020004e00 con 0x7f902c104bb0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.943+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c19fea0 con 0x7f902c104bb0 2024-04-24T23:48:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.944+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c0fd720 con 0x7f902c104bb0 2024-04-24T23:48:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.945+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9020004fa0 con 0x7f902c104bb0 2024-04-24T23:48:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.945+0000 7f90297fa640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 0x7f90140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.945+0000 7f902bfff640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 0x7f90140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.945+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9638+0+0 (secure 0 0 0) 0x7f90200cef90 con 0x7f902c104bb0 2024-04-24T23:48:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.946+0000 7f902bfff640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 0x7f90140a5050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f901c00bbe0 tx=0x7f901c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:47.950+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f902009b970 con 0x7f902c104bb0 2024-04-24T23:48:48.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a[29114]: 2024-04-24T23:48:47.819+0000 7fd0f68e6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-mds[68395]: -- 172.21.15.107:0/2732587890 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55867b205a40 con 0x55867b1d8800 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: starting mds.g at 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-mds[68754]: -- 172.21.15.107:0/2984505760 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a05c6fa40 con 0x556a05c42c00 2024-04-24T23:48:48.128 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: starting mds.a at 2024-04-24T23:48:48.129 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:48:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: 2024-04-24T23:48:47.864+0000 7fc5ce50e700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6838/0: (98) Address already in use 2024-04-24T23:48:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.165+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f902c062340 con 0x7f902c104bb0 2024-04-24T23:48:48.177 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.177 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.177 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-mds[63070]: -- 172.21.15.170:0/1830479556 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ff4e1da40 con 0x560ff4df0c00 2024-04-24T23:48:48.177 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: starting mds.b at 2024-04-24T23:48:48.177 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-mds[62110]: -- 172.21.15.170:0/1423268833 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590e1c0ba40 con 0x5590e1bde800 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: starting mds.k at 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-mds[62434]: -- 172.21.15.170:0/2922888472 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56055c2a1a40 con 0x56055c275000 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: starting mds.e at 2024-04-24T23:48:48.178 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:48:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: starting mds.h at 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mds[63266]: -- 172.21.15.195:0/3990095871 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b2bb45a40 con 0x555b2bb18800 2024-04-24T23:48:48.179 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: starting mds.f at 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mds[62952]: -- 172.21.15.195:0/390115792 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c743405a40 con 0x55c7433d9000 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: starting mds.l at 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mds[63890]: -- 172.21.15.195:0/2578822443 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3ac401a40 con 0x55f3ac3d4c00 2024-04-24T23:48:48.180 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: starting mds.i at 2024-04-24T23:48:48.181 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: ignoring --setuser ceph since I am not root 2024-04-24T23:48:48.181 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: ignoring --setgroup ceph since I am not root 2024-04-24T23:48:48.181 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-mds[63571]: -- 172.21.15.195:0/1414138027 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db8ee23a40 con 0x55db8edf7000 2024-04-24T23:48:48.181 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:48:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: starting mds.c at 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3553305901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3014130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-24T23:48:48.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:48 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3553305901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3014130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:48:48.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T23:48:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T23:48:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-24T23:48:48.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:48 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:48:48.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3553305901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3014130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T23:48:48.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2970475080' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-24T23:48:48.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:48 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T23:48:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.822+0000 7f90297fa640 1 -- 172.21.15.107:0/2918642015 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) ==== 104+0+0 (secure 0 0 0) 0x7f902009b270 con 0x7f902c104bb0 2024-04-24T23:48:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.825+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 msgr2=0x7f90140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.825+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 0x7f90140a5050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f901c00bbe0 tx=0x7f901c00b390 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.825+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 msgr2=0x7f902c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.825+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f902000d740 tx=0x7f902000dc00 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 shutdown_connections 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f90140a2b90 0x7f90140a5050 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f902c1074e0 0x7f902c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c104bb0 0x7f902c198040 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 --2- 172.21.15.107:0/2918642015 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f902c102280 0x7f902c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 >> 172.21.15.107:0/2918642015 conn(0x7f902c0fbc30 msgr2=0x7f902c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 shutdown_connections 2024-04-24T23:48:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.826+0000 7f9032838640 1 -- 172.21.15.107:0/2918642015 wait complete. 2024-04-24T23:48:48.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T23:48:48.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 -- 172.21.15.107:0/3648899986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88107460 msgr2=0x7f1e881098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3648899986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88107460 0x7f1e881098b0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009ca0 tx=0x7f1e8002a8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 -- 172.21.15.107:0/3648899986 shutdown_connections 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3648899986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88107460 0x7f1e881098b0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3648899986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88104b30 0x7f1e88106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3648899986 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88102200 0x7f1e881045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 -- 172.21.15.107:0/3648899986 >> 172.21.15.107:0/3648899986 conn(0x7f1e880fbbf0 msgr2=0x7f1e880fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 -- 172.21.15.107:0/3648899986 shutdown_connections 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.949+0000 7f1e900a5640 1 -- 172.21.15.107:0/3648899986 wait complete. 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 Processor -- start 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 -- start start 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 0x7f1e88197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 0x7f1e8819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8810d400 con 0x7f1e88104b30 2024-04-24T23:48:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8810d280 con 0x7f1e88102200 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e900a5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8810d580 con 0x7f1e88107460 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46206/0 (socket says 172.21.15.107:46206) 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 -- 172.21.15.107:0/3294539835 learned_addr learned my addr 172.21.15.107:0/3294539835 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8e642640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 0x7f1e8819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8de41640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 0x7f1e88197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 -- 172.21.15.107:0/3294539835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 msgr2=0x7f1e8819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 0x7f1e8819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 -- 172.21.15.107:0/3294539835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 msgr2=0x7f1e88197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 0x7f1e88197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8d640640 1 -- 172.21.15.107:0/3294539835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8819f6c0 con 0x7f1e88104b30 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8e642640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 0x7f1e8819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.950+0000 7f1e8de41640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 0x7f1e88197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e8d640640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800d770 tx=0x7f1e7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78017070 con 0x7f1e88104b30 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8819f9b0 con 0x7f1e88104b30 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e78009c30 con 0x7f1e88104b30 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800f620 con 0x7f1e88104b30 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.951+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e8819ff00 con 0x7f1e88104b30 2024-04-24T23:48:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.952+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e880fd6e0 con 0x7f1e88104b30 2024-04-24T23:48:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.953+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1e7801a070 con 0x7f1e88104b30 2024-04-24T23:48:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.954+0000 7f1e7effd640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 0x7f1e740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.954+0000 7f1e8de41640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 0x7f1e740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.954+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9638+0+0 (secure 0 0 0) 0x7f1e780cf170 con 0x7f1e88104b30 2024-04-24T23:48:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.954+0000 7f1e8de41640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 0x7f1e740a5050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1e840058c0 tx=0x7f1e84005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:48.957+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e78009de0 con 0x7f1e88104b30 2024-04-24T23:48:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.135+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1e8802d170 con 0x7f1e88104b30 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: pgmap v474: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: Cluster is now healthy 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:48:49.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6838/1747141538,v1:172.21.15.107:6840/1747141538] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.107:6839/2814209986,v1:172.21.15.107:6841/2814209986] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6833/1899795844,v1:172.21.15.195:6836/1899795844] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6832/2021016383,v1:172.21.15.195:6835/2021016383] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6834/2423999816,v1:172.21.15.195:6837/2423999816] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.195:6838/1875121780,v1:172.21.15.195:6839/1875121780] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6832/1755472105,v1:172.21.15.170:6834/1755472105] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6833/2515971920,v1:172.21.15.170:6835/2515971920] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6836/3783749945,v1:172.21.15.170:6838/3783749945] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: mds.? [v2:172.21.15.170:6837/2312680997,v1:172.21.15.170:6839/2312680997] up:boot 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: fsmap 12 up:standby 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:48:49.563 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:48:49.564 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:49 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:49.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: pgmap v474: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: Cluster is now healthy 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6838/1747141538,v1:172.21.15.107:6840/1747141538] up:boot 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.107:6839/2814209986,v1:172.21.15.107:6841/2814209986] up:boot 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6833/1899795844,v1:172.21.15.195:6836/1899795844] up:boot 2024-04-24T23:48:49.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6832/2021016383,v1:172.21.15.195:6835/2021016383] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6834/2423999816,v1:172.21.15.195:6837/2423999816] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.195:6838/1875121780,v1:172.21.15.195:6839/1875121780] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6832/1755472105,v1:172.21.15.170:6834/1755472105] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6833/2515971920,v1:172.21.15.170:6835/2515971920] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6836/3783749945,v1:172.21.15.170:6838/3783749945] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: mds.? [v2:172.21.15.170:6837/2312680997,v1:172.21.15.170:6839/2312680997] up:boot 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: fsmap 12 up:standby 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:48:49.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:48:49.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:49 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: pgmap v474: 129 pgs: 129 active+clean; 9.1 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: Cluster is now healthy 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2918642015' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6838/1747141538,v1:172.21.15.107:6840/1747141538] up:boot 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.107:6839/2814209986,v1:172.21.15.107:6841/2814209986] up:boot 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6833/1899795844,v1:172.21.15.195:6836/1899795844] up:boot 2024-04-24T23:48:49.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6832/2021016383,v1:172.21.15.195:6835/2021016383] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6834/2423999816,v1:172.21.15.195:6837/2423999816] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.195:6838/1875121780,v1:172.21.15.195:6839/1875121780] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6832/1755472105,v1:172.21.15.170:6834/1755472105] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6833/2515971920,v1:172.21.15.170:6835/2515971920] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6836/3783749945,v1:172.21.15.170:6838/3783749945] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: mds.? [v2:172.21.15.170:6837/2312680997,v1:172.21.15.170:6839/2312680997] up:boot 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: fsmap 12 up:standby 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T23:48:49.629 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T23:48:49.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T23:48:49.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='mgr.14152 172.21.15.107:0/3460042676' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T23:48:49.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:49 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.831+0000 7f1e7effd640 1 -- 172.21.15.107:0/3294539835 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f1e7809bb50 con 0x7f1e88104b30 2024-04-24T23:48:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 msgr2=0x7f1e740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 0x7f1e740a5050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1e840058c0 tx=0x7f1e84005d30 comp rx=0 tx=0).stop 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 msgr2=0x7f1e88197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800d770 tx=0x7f1e7800dc30 comp rx=0 tx=0).stop 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 shutdown_connections 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f1e740a2b90 0x7f1e740a5050 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f1e88107460 0x7f1e8819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e88104b30 0x7f1e88197fe0 secure :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800d770 tx=0x7f1e7800dc30 comp rx=0 tx=0).stop 2024-04-24T23:48:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 --2- 172.21.15.107:0/3294539835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e88102200 0x7f1e88197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 >> 172.21.15.107:0/3294539835 conn(0x7f1e880fbbf0 msgr2=0x7f1e88102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 shutdown_connections 2024-04-24T23:48:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.835+0000 7f1e900a5640 1 -- 172.21.15.107:0/3294539835 wait complete. 2024-04-24T23:48:49.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T23:48:49.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- 172.21.15.107:0/147703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 msgr2=0x7fda64101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 --2- 172.21.15.107:0/147703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda64101d10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fda50009900 tx=0x7fda5002a410 comp rx=0 tx=0).stop 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- 172.21.15.107:0/147703718 shutdown_connections 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 --2- 172.21.15.107:0/147703718 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 0x7fda64108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 --2- 172.21.15.107:0/147703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda64101d10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 --2- 172.21.15.107:0/147703718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 0x7fda64100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- 172.21.15.107:0/147703718 >> 172.21.15.107:0/147703718 conn(0x7fda640fbc10 msgr2=0x7fda640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- 172.21.15.107:0/147703718 shutdown_connections 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- 172.21.15.107:0/147703718 wait complete. 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 Processor -- start 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.953+0000 7fda6aca5640 1 -- start start 2024-04-24T23:48:49.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 0x7fda64197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 0x7fda6419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410c380 con 0x7fda64101890 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410c200 con 0x7fda641006a0 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410c500 con 0x7fda64102250 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46228/0 (socket says 172.21.15.107:46228) 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 -- 172.21.15.107:0/1349363683 learned_addr learned my addr 172.21.15.107:0/1349363683 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda68a41640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 0x7fda64197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda69242640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 0x7fda6419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 -- 172.21.15.107:0/1349363683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 msgr2=0x7fda6419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 0x7fda6419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 -- 172.21.15.107:0/1349363683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 msgr2=0x7fda64197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 0x7fda64197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 -- 172.21.15.107:0/1349363683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6419f780 con 0x7fda64101890 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda63fff640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fda50009230 tx=0x7fda50004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5003f030 con 0x7fda64101890 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6419fa10 con 0x7fda64101890 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda50003ce0 con 0x7fda64101890 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.954+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50035070 con 0x7fda64101890 2024-04-24T23:48:49.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.955+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6419ff80 con 0x7fda64101890 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.956+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda641a0150 con 0x7fda64101890 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.957+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fda50042030 con 0x7fda64101890 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.957+0000 7fda61ffb640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 0x7fda540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.957+0000 7fda68a41640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 0x7fda540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.957+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9084+0+0 (secure 0 0 0) 0x7fda500efe40 con 0x7fda64101890 2024-04-24T23:48:49.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.958+0000 7fda68a41640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 0x7fda540a5050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fda641016f0 tx=0x7fda5800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:49.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:49.961+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda500bc400 con 0x7fda64101890 2024-04-24T23:48:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.139+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fda641a0320 con 0x7fda64101890 2024-04-24T23:48:50.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.833+0000 7fda61ffb640 1 -- 172.21.15.107:0/1349363683 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7fda500c0700 con 0x7fda64101890 2024-04-24T23:48:50.833 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2024-04-24T23:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.836+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 msgr2=0x7fda540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.836+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 0x7fda540a5050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fda641016f0 tx=0x7fda5800a2d0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.836+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 msgr2=0x7fda641980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.836+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fda50009230 tx=0x7fda50004430 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 shutdown_connections 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7fda540a2b90 0x7fda540a5050 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fda64102250 0x7fda6419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda64101890 0x7fda641980a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 --2- 172.21.15.107:0/1349363683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda641006a0 0x7fda64197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 >> 172.21.15.107:0/1349363683 conn(0x7fda640fbc10 msgr2=0x7fda640fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 shutdown_connections 2024-04-24T23:48:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.837+0000 7fda6aca5640 1 -- 172.21.15.107:0/1349363683 wait complete. 2024-04-24T23:48:50.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- 172.21.15.107:0/3175655724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 msgr2=0x7f4614100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 --2- 172.21.15.107:0/3175655724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614100ac0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4608009900 tx=0x7f460802a410 comp rx=0 tx=0).stop 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- 172.21.15.107:0/3175655724 shutdown_connections 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 --2- 172.21.15.107:0/3175655724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614102250 0x7f4614108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 --2- 172.21.15.107:0/3175655724 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614101890 0x7f4614101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 --2- 172.21.15.107:0/3175655724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614100ac0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- 172.21.15.107:0/3175655724 >> 172.21.15.107:0/3175655724 conn(0x7f46140fbc10 msgr2=0x7f46140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- 172.21.15.107:0/3175655724 shutdown_connections 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- 172.21.15.107:0/3175655724 wait complete. 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 Processor -- start 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.957+0000 7f461af10640 1 -- start start 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614101890 0x7f46141981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614102250 0x7f461419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f461410d500 con 0x7f46141006a0 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f461410d380 con 0x7f4614101890 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 -- 0x7f461410d680 con 0x7f4614102250 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46248/0 (socket says 172.21.15.107:46248) 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 -- 172.21.15.107:0/2580981077 learned_addr learned my addr 172.21.15.107:0/2580981077 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-04-24T23:48:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4613fff640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614101890 0x7f46141981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f46194ad640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614102250 0x7f461419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 -- 172.21.15.107:0/2580981077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614102250 msgr2=0x7f461419f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614102250 0x7f461419f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 -- 172.21.15.107:0/2580981077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614101890 msgr2=0x7f46141981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614101890 0x7f46141981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 -- 172.21.15.107:0/2580981077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461419f890 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4618cac640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4608031610 tx=0x7f46080043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460803f030 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608003ce0 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461419fb20 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.958+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608035070 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.959+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46141a0010 con 0x7f46141006a0 2024-04-24T23:48:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.959+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4614107b30 con 0x7f46141006a0 2024-04-24T23:48:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.961+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4608042030 con 0x7f46141006a0 2024-04-24T23:48:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.961+0000 7f4611ffb640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 0x7f46040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.961+0000 7f4613fff640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 0x7f46040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.961+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8496+0+0 (secure 0 0 0) 0x7f46080efb20 con 0x7f46141006a0 2024-04-24T23:48:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.962+0000 7f4613fff640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 0x7f46040a5050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4600009890 tx=0x7f4600008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:50.965+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f46080bc980 con 0x7f46141006a0 2024-04-24T23:48:51.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:50 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:51.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:50 smithi107 ceph-mon[29118]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T23:48:51.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:50 smithi107 ceph-mon[29118]: pgmap v476: 97 pgs: 97 active+clean; 577 KiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:48:51.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:50 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:51.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.149+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f461402d290 con 0x7f46141006a0 2024-04-24T23:48:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:50 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:50 smithi195 ceph-mon[33019]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T23:48:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:50 smithi195 ceph-mon[33019]: pgmap v476: 97 pgs: 97 active+clean; 577 KiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:48:51.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:50 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:50 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/3294539835' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:50 smithi170 ceph-mon[32714]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T23:48:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:50 smithi170 ceph-mon[32714]: pgmap v476: 97 pgs: 97 active+clean; 577 KiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T23:48:51.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:50 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:51.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.844+0000 7f4611ffb640 1 -- 172.21.15.107:0/2580981077 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f46080bc280 con 0x7f46141006a0 2024-04-24T23:48:51.844 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data_ec' removed 2024-04-24T23:48:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 msgr2=0x7f46040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 0x7f46040a5050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4600009890 tx=0x7f4600008040 comp rx=0 tx=0).stop 2024-04-24T23:48:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 msgr2=0x7f4614197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4608031610 tx=0x7f46080043b0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 shutdown_connections 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:6800/1159279165,v1:172.21.15.107:6801/1159279165] conn(0x7f46040a2b90 0x7f46040a5050 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4614102250 0x7f461419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4614101890 0x7f46141981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 --2- 172.21.15.107:0/2580981077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46141006a0 0x7f4614197c80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.847+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 >> 172.21.15.107:0/2580981077 conn(0x7f46140fbc10 msgr2=0x7f4614104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.848+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 shutdown_connections 2024-04-24T23:48:51.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-24T23:48:51.848+0000 7f461af10640 1 -- 172.21.15.107:0/2580981077 wait complete. 2024-04-24T23:48:51.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:48:51.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:48:51.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:48:51.955 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-24T23:48:51.956 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T23:48:51.971 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T23:48:51.986 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T23:48:52.012 INFO:tasks.cephadm:Stopping all daemons... 2024-04-24T23:48:52.012 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-24T23:48:52.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a 2024-04-24T23:48:52.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:51 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:52.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:51 smithi107 ceph-mon[29118]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T23:48:52.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:51 smithi107 ceph-mon[29118]: from='client.? 172.21.15.107:0/2580981077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:52.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:52 smithi107 systemd[1]: Stopping Ceph mon.a for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:52.311 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:51 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:51 smithi195 ceph-mon[33019]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T23:48:52.312 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:51 smithi195 ceph-mon[33019]: from='client.? 172.21.15.107:0/2580981077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:51 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/1349363683' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T23:48:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:51 smithi170 ceph-mon[32714]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T23:48:52.335 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:51 smithi170 ceph-mon[32714]: from='client.? 172.21.15.107:0/2580981077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T23:48:52.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a[29114]: 2024-04-24T23:48:52.342+0000 7fd0fc0f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:48:52.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a[29114]: 2024-04-24T23:48:52.342+0000 7fd0fc0f1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-24T23:48:52.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:52 smithi107 podman[87708]: 2024-04-24 23:48:52.516655292 +0000 UTC m=+0.305154768 container died 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-24T23:48:53.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:53 smithi107 podman[87708]: 2024-04-24 23:48:53.019204321 +0000 UTC m=+0.807703803 container cleanup 14250468e610de09581421d3ea58b274274b8d36a167f6a5b0444d9e170cdba8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:48:53.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Apr 24 23:48:53 smithi107 bash[87708]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-a 2024-04-24T23:48:53.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.a.service' 2024-04-24T23:48:53.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:48:53.561 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-24T23:48:53.561 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-24T23:48:53.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.b 2024-04-24T23:48:53.868 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:53 smithi170 systemd[1]: Stopping Ceph mon.b for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:54.261 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:53 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-b[32710]: 2024-04-24T23:48:53.868+0000 7fdf5e14a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:48:54.261 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:53 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-b[32710]: 2024-04-24T23:48:53.868+0000 7fdf5e14a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-24T23:48:54.261 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:53 smithi170 podman[66144]: 2024-04-24 23:48:53.971188471 +0000 UTC m=+0.215823617 container died 0dbe7c606142af67d0f5087421d0028459cd33692bcb475e3b378f6e0b5659be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T23:48:54.585 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:54 smithi170 podman[66144]: 2024-04-24 23:48:54.369810729 +0000 UTC m=+0.614445880 container cleanup 0dbe7c606142af67d0f5087421d0028459cd33692bcb475e3b378f6e0b5659be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-b, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:48:54.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Apr 24 23:48:54 smithi170 bash[66144]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-b 2024-04-24T23:48:54.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.b.service' 2024-04-24T23:48:54.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:48:54.882 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-24T23:48:54.882 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-24T23:48:54.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.c 2024-04-24T23:48:55.188 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:54 smithi195 systemd[1]: Stopping Ceph mon.c for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c[33000]: 2024-04-24T23:48:55.187+0000 7f3e1c6fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:48:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c[33000]: 2024-04-24T23:48:55.187+0000 7f3e1c6fd700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-24T23:48:55.562 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:55 smithi195 podman[67021]: 2024-04-24 23:48:55.423257677 +0000 UTC m=+0.348231709 container died 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux ) 2024-04-24T23:48:55.989 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:55 smithi195 podman[67021]: 2024-04-24 23:48:55.788476879 +0000 UTC m=+0.713450909 container cleanup 43ab3efaf3195f14131a95f2d2c365a2bae3074045299e4d35622e5e9879667c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-24T23:48:55.989 INFO:journalctl@ceph.mon.c.smithi195.stdout:Apr 24 23:48:55 smithi195 bash[67021]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mon-c 2024-04-24T23:48:56.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mon.c.service' 2024-04-24T23:48:56.255 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:48:56.256 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-24T23:48:56.256 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-24T23:48:56.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x 2024-04-24T23:48:56.627 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:56 smithi107 systemd[1]: Stopping Ceph mgr.x for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:56.967 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:56 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x[29352]: 2024-04-24T23:48:56.642+0000 7f45917e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:48:56.968 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:56 smithi107 podman[87832]: 2024-04-24 23:48:56.673696534 +0000 UTC m=+0.224378625 container died b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:48:57.627 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:57 smithi107 podman[87832]: 2024-04-24 23:48:57.243457382 +0000 UTC m=+0.794139465 container cleanup b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T23:48:57.627 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:57 smithi107 bash[87832]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x 2024-04-24T23:48:57.628 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Apr 24 23:48:57 smithi107 podman[87846]: 2024-04-24 23:48:57.453831738 +0000 UTC m=+0.778795787 container remove b0472117974e0bef86bf87617ba807d5d082d708a6dcb59dbe5a4465a09ad8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-24T23:48:57.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-24T23:48:57.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:48:57.691 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-24T23:48:57.691 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-24T23:48:57.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.y 2024-04-24T23:48:58.028 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:48:57 smithi170 systemd[1]: Stopping Ceph mgr.y for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:58.335 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:48:58 smithi170 podman[66267]: 2024-04-24 23:48:58.029298356 +0000 UTC m=+0.142546608 container died 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-24T23:48:58.701 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:48:58 smithi170 podman[66267]: 2024-04-24 23:48:58.434010491 +0000 UTC m=+0.547258743 container cleanup 081385a7b466381e56a100524f2b14938f0ef57b43add57d57e0f1e3d1c839aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:48:58.701 INFO:journalctl@ceph.mgr.y.smithi170.stdout:Apr 24 23:48:58 smithi170 bash[66267]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-y 2024-04-24T23:48:58.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-24T23:48:58.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:48:58.876 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-24T23:48:58.876 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-24T23:48:58.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.z 2024-04-24T23:48:59.221 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:48:58 smithi195 systemd[1]: Stopping Ceph mgr.z for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:48:59.485 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:48:59 smithi195 podman[67142]: 2024-04-24 23:48:59.22298822 +0000 UTC m=+0.149670080 container died cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:48:59.811 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:48:59 smithi195 podman[67142]: 2024-04-24 23:48:59.610805403 +0000 UTC m=+0.537487274 container cleanup cb54a4c4c63bc27e248373aee2dc52cf880540ccaa9ca54c2415a089d11b2d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-24T23:48:59.812 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Apr 24 23:48:59 smithi195 bash[67142]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mgr-z 2024-04-24T23:49:00.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-24T23:49:00.116 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:00.117 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-24T23:49:00.117 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-24T23:49:00.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.0 2024-04-24T23:49:00.413 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:00 smithi107 systemd[1]: Stopping Ceph osd.0 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:00.878 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0[41364]: 2024-04-24T23:49:00.414+0000 7f3b9d878700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:00.879 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0[41364]: 2024-04-24T23:49:00.414+0000 7f3b9d878700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-24T23:49:00.879 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0[41364]: 2024-04-24T23:49:00.414+0000 7f3b9d878700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:06.056 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:05 smithi107 podman[87954]: 2024-04-24 23:49:05.771599652 +0000 UTC m=+5.462411020 container died ff810b2e0994a0cf9b5fb08709bee8983f003d7175ba1743e67b0f13e8e09eed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T23:49:06.377 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:06 smithi107 podman[87954]: 2024-04-24 23:49:06.240242658 +0000 UTC m=+5.931054090 container cleanup ff810b2e0994a0cf9b5fb08709bee8983f003d7175ba1743e67b0f13e8e09eed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-24T23:49:06.600 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:06 smithi107 bash[87954]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0 2024-04-24T23:49:06.849 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:06 smithi107 podman[87972]: 2024-04-24 23:49:06.525038438 +0000 UTC m=+0.749197482 container remove ff810b2e0994a0cf9b5fb08709bee8983f003d7175ba1743e67b0f13e8e09eed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-24T23:49:07.127 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:06 smithi107 podman[88064]: 2024-04-24 23:49:06.850862056 +0000 UTC m=+0.112618504 container create 5f03e4827aec4745747334076ff610f697e2ff0fe8cb488f621c24c9587a1ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T23:49:07.128 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:06 smithi107 podman[88064]: 2024-04-24 23:49:06.756146842 +0000 UTC m=+0.017903306 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:07.395 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:07 smithi107 podman[88064]: 2024-04-24 23:49:07.137328581 +0000 UTC m=+0.399085030 container init 5f03e4827aec4745747334076ff610f697e2ff0fe8cb488f621c24c9587a1ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T23:49:07.395 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:07 smithi107 podman[88064]: 2024-04-24 23:49:07.140904182 +0000 UTC m=+0.402660632 container start 5f03e4827aec4745747334076ff610f697e2ff0fe8cb488f621c24c9587a1ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-24T23:49:07.395 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:07 smithi107 podman[88064]: 2024-04-24 23:49:07.396271948 +0000 UTC m=+0.658028400 container attach 5f03e4827aec4745747334076ff610f697e2ff0fe8cb488f621c24c9587a1ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-24T23:49:07.798 INFO:journalctl@ceph.osd.0.smithi107.stdout:Apr 24 23:49:07 smithi107 podman[88064]: 2024-04-24 23:49:07.449547828 +0000 UTC m=+0.711304278 container died 5f03e4827aec4745747334076ff610f697e2ff0fe8cb488f621c24c9587a1ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-24T23:49:08.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.0.service' 2024-04-24T23:49:08.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:08.155 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-24T23:49:08.155 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-24T23:49:08.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.1 2024-04-24T23:49:08.472 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:08 smithi170 systemd[1]: Stopping Ceph osd.1 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:08.835 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1[37311]: 2024-04-24T23:49:08.471+0000 7f8751078700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:08.835 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1[37311]: 2024-04-24T23:49:08.471+0000 7f8751078700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-24T23:49:08.835 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1[37311]: 2024-04-24T23:49:08.471+0000 7f8751078700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:14.297 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:13 smithi170 podman[66403]: 2024-04-24 23:49:13.995528693 +0000 UTC m=+5.657724878 container died a8b1564bb50020f43dbc489a7aa72b6f0bfe6400dd2993c9368ae25eb7937e3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-24T23:49:14.585 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:14 smithi170 podman[66403]: 2024-04-24 23:49:14.397406107 +0000 UTC m=+6.059602290 container cleanup a8b1564bb50020f43dbc489a7aa72b6f0bfe6400dd2993c9368ae25eb7937e3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T23:49:14.585 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:14 smithi170 bash[66403]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1 2024-04-24T23:49:15.014 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:14 smithi170 podman[66421]: 2024-04-24 23:49:14.665524776 +0000 UTC m=+0.659287192 container remove a8b1564bb50020f43dbc489a7aa72b6f0bfe6400dd2993c9368ae25eb7937e3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T23:49:15.268 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:14.919011013 +0000 UTC m=+0.018116839 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:15.268 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:15.053417947 +0000 UTC m=+0.152523764 container create aeac7e844246c9ba3cbe4f91b86d452c62bcfc1620e74558f0e250915a782bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-24T23:49:15.268 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:15.269360783 +0000 UTC m=+0.368466596 container init aeac7e844246c9ba3cbe4f91b86d452c62bcfc1620e74558f0e250915a782bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T23:49:15.556 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:15.272836475 +0000 UTC m=+0.371942290 container start aeac7e844246c9ba3cbe4f91b86d452c62bcfc1620e74558f0e250915a782bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-24T23:49:15.557 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:15.361067805 +0000 UTC m=+0.460173625 container attach aeac7e844246c9ba3cbe4f91b86d452c62bcfc1620e74558f0e250915a782bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-24T23:49:15.835 INFO:journalctl@ceph.osd.1.smithi170.stdout:Apr 24 23:49:15 smithi170 podman[66513]: 2024-04-24 23:49:15.557983371 +0000 UTC m=+0.657089190 container died aeac7e844246c9ba3cbe4f91b86d452c62bcfc1620e74558f0e250915a782bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-24T23:49:16.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.1.service' 2024-04-24T23:49:16.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:16.284 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-24T23:49:16.284 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-24T23:49:16.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.2 2024-04-24T23:49:16.562 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:16 smithi195 systemd[1]: Stopping Ceph osd.2 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:17.062 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2[38138]: 2024-04-24T23:49:16.599+0000 7f3cf7724700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:17.062 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2[38138]: 2024-04-24T23:49:16.599+0000 7f3cf7724700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-24T23:49:17.062 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2[38138]: 2024-04-24T23:49:16.599+0000 7f3cf7724700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:22.315 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:22 smithi195 podman[67279]: 2024-04-24 23:49:22.039532183 +0000 UTC m=+5.564873751 container died 19e9ab696facc45c94f59cf3a2e838c23cce99e45b89a9ec336ae0708beb9190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T23:49:22.707 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:22 smithi195 podman[67279]: 2024-04-24 23:49:22.41565053 +0000 UTC m=+5.940992089 container cleanup 19e9ab696facc45c94f59cf3a2e838c23cce99e45b89a9ec336ae0708beb9190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:49:22.708 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:22 smithi195 bash[67279]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2 2024-04-24T23:49:23.001 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:22 smithi195 podman[67296]: 2024-04-24 23:49:22.708720417 +0000 UTC m=+0.668088517 container remove 19e9ab696facc45c94f59cf3a2e838c23cce99e45b89a9ec336ae0708beb9190 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:49:23.296 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:22.904922383 +0000 UTC m=+0.017763780 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:23.296 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:23.026497041 +0000 UTC m=+0.139338422 container create 464672166de3039533addff0f7c782b1ab1fd57510869551d6d87a8ddf5b93a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:23.296 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:23.296231774 +0000 UTC m=+0.409073153 container init 464672166de3039533addff0f7c782b1ab1fd57510869551d6d87a8ddf5b93a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T23:49:23.562 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:23.299835618 +0000 UTC m=+0.412676997 container start 464672166de3039533addff0f7c782b1ab1fd57510869551d6d87a8ddf5b93a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T23:49:23.562 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:23.421301123 +0000 UTC m=+0.534142511 container attach 464672166de3039533addff0f7c782b1ab1fd57510869551d6d87a8ddf5b93a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-24T23:49:23.890 INFO:journalctl@ceph.osd.2.smithi195.stdout:Apr 24 23:49:23 smithi195 podman[67389]: 2024-04-24 23:49:23.591130364 +0000 UTC m=+0.703971736 container died 464672166de3039533addff0f7c782b1ab1fd57510869551d6d87a8ddf5b93a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:49:24.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.2.service' 2024-04-24T23:49:24.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:24.270 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-24T23:49:24.270 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-24T23:49:24.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.3 2024-04-24T23:49:24.584 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:24 smithi107 systemd[1]: Stopping Ceph osd.3 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:24.844 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:24 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:24.584+0000 7f356f53a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:24.844 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:24 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:24.584+0000 7f356f53a700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-24T23:49:24.845 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:24 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:24.584+0000 7f356f53a700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:27.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:27.192+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:27.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:27.017+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:27.877 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:27 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:27.546+0000 7f3567b26700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:05.402772+0000 front 2024-04-24T23:49:05.402648+0000 (oldest deadline 2024-04-24T23:49:27.102811+0000) 2024-04-24T23:49:28.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:28.217+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:28.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:27.977+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:28.877 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:28 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:28.522+0000 7f3567b26700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:05.402772+0000 front 2024-04-24T23:49:05.402648+0000 (oldest deadline 2024-04-24T23:49:27.102811+0000) 2024-04-24T23:49:29.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:28 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:28.811+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:29.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:29.193+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:29.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:28.995+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:29.877 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:29 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3[48191]: 2024-04-24T23:49:29.539+0000 7f3567b26700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:05.402772+0000 front 2024-04-24T23:49:05.402648+0000 (oldest deadline 2024-04-24T23:49:27.102811+0000) 2024-04-24T23:49:30.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:29 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:29.816+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:30.210 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:29 smithi107 podman[88175]: 2024-04-24 23:49:29.911997757 +0000 UTC m=+5.447519209 container died 8734e2be7ab1e02f3f422509284c61b79cb88339e8da2f6ab84f425695b66be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:30.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:30.153+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:30.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:30.027+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:30.627 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:30 smithi107 podman[88175]: 2024-04-24 23:49:30.418730686 +0000 UTC m=+5.954252149 container cleanup 8734e2be7ab1e02f3f422509284c61b79cb88339e8da2f6ab84f425695b66be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:30.628 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:30 smithi107 bash[88175]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3 2024-04-24T23:49:31.061 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:30 smithi107 podman[88204]: 2024-04-24 23:49:30.645198261 +0000 UTC m=+0.730582023 container remove 8734e2be7ab1e02f3f422509284c61b79cb88339e8da2f6ab84f425695b66be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T23:49:31.311 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:31.132+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:31.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:31.009+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:31.323 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:30.965639875 +0000 UTC m=+0.017957130 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:31.323 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:31.07946307 +0000 UTC m=+0.131780318 container create defecb68c5917efe218c9275cdb457aa1605c0745c4fe9bb6873bdab9f509d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-24T23:49:31.323 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:31.324332178 +0000 UTC m=+0.376649423 container init defecb68c5917efe218c9275cdb457aa1605c0745c4fe9bb6873bdab9f509d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-24T23:49:31.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:30 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:30.847+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:31.619 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:31.327732854 +0000 UTC m=+0.380050097 container start defecb68c5917efe218c9275cdb457aa1605c0745c4fe9bb6873bdab9f509d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T23:49:31.619 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:31.415881938 +0000 UTC m=+0.468199187 container attach defecb68c5917efe218c9275cdb457aa1605c0745c4fe9bb6873bdab9f509d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:49:31.877 INFO:journalctl@ceph.osd.3.smithi107.stdout:Apr 24 23:49:31 smithi107 podman[88295]: 2024-04-24 23:49:31.620386084 +0000 UTC m=+0.672703331 container died defecb68c5917efe218c9275cdb457aa1605c0745c4fe9bb6873bdab9f509d1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:49:32.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:32.179+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:32.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:31.998+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:32.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:31 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:31.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:32.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.3.service' 2024-04-24T23:49:32.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:32.398 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-24T23:49:32.398 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-24T23:49:32.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.4 2024-04-24T23:49:32.708 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:32 smithi170 systemd[1]: Stopping Ceph osd.4 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:33.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:32 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:32.926+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:33.085 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:32 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4[43821]: 2024-04-24T23:49:32.708+0000 7f5a4298a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:33.086 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:32 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4[43821]: 2024-04-24T23:49:32.708+0000 7f5a4298a700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-24T23:49:33.086 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:32 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4[43821]: 2024-04-24T23:49:32.708+0000 7f5a4298a700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:33.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:33.140+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:33.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:32.965+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:34.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:34 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:34.101+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:34.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:33.991+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:34.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:33 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:33.973+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:35.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:35 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:35.134+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:35.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:35 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:35.037+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:35.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:35 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:35.016+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:36.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:36 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:36.010+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:36.296 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:36 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:36.010+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:36.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:36 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:36.169+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:36.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:36 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:36.257+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:36.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:36 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:36.067+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:36.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:36 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:36.297+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:37.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:37 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:37.165+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:37.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:37 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:37.257+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:37.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:37 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:37.032+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:37.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:37 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:37.032+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:37.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:37 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:37.287+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:37.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:37 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:37.046+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:37.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:37 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:37.046+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:38.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:37 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:37.923+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:38.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:37 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:37.755+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:38.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:38.267+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:38.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:38.018+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:38.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:38.018+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:38.313 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:38.177+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:38.313 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:38 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:38.177+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:38.475 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:38 smithi170 podman[66617]: 2024-04-24 23:49:38.207350398 +0000 UTC m=+5.612386673 container died 1710d2c12be76ab32c8170b6aea152ecaefae2c0683c0a2bf111f597e7ba8dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T23:49:38.476 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:38 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:38.303+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:38.476 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:38 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:38.093+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:38.476 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:38 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:38.093+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:38.835 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:38 smithi170 podman[66617]: 2024-04-24 23:49:38.576195184 +0000 UTC m=+5.981231465 container cleanup 1710d2c12be76ab32c8170b6aea152ecaefae2c0683c0a2bf111f597e7ba8dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-24T23:49:38.835 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:38 smithi170 bash[66617]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4 2024-04-24T23:49:39.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:38 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:38.903+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:39.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:38 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:38.708+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:39.238 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:38 smithi170 podman[66645]: 2024-04-24 23:49:38.886015438 +0000 UTC m=+0.675940708 container remove 1710d2c12be76ab32c8170b6aea152ecaefae2c0683c0a2bf111f597e7ba8dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-24T23:49:39.239 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:39 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:39.072+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:39.239 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:39 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:39.072+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:39.555 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:39 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:39.287+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:39.555 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.143222825 +0000 UTC m=+0.017759497 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:39.555 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.282310598 +0000 UTC m=+0.156847262 container create 60b050348f57c326c35fb9942d0cff659230ce556bd8bd63c31c529c26ebb222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-24T23:49:39.561 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:39 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:39.281+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:39.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:39 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:39.066+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:39.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:39 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:39.066+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:39.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:39 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:39.211+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:39.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:39 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:39.211+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:39.835 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.556519715 +0000 UTC m=+0.431056379 container init 60b050348f57c326c35fb9942d0cff659230ce556bd8bd63c31c529c26ebb222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:49:39.835 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.559688563 +0000 UTC m=+0.434225227 container start 60b050348f57c326c35fb9942d0cff659230ce556bd8bd63c31c529c26ebb222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-24T23:49:39.835 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.631303288 +0000 UTC m=+0.505839988 container attach 60b050348f57c326c35fb9942d0cff659230ce556bd8bd63c31c529c26ebb222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-24T23:49:40.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:39 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:39.858+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:40.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:39 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:39.700+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:40.200 INFO:journalctl@ceph.osd.4.smithi170.stdout:Apr 24 23:49:39 smithi170 podman[66738]: 2024-04-24 23:49:39.844916126 +0000 UTC m=+0.719452791 container died 60b050348f57c326c35fb9942d0cff659230ce556bd8bd63c31c529c26ebb222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:49:40.200 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:40 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:40.082+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:40.200 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:40 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:40.082+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:40.311 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:40.247+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:40.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:40.053+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:40.312 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:40.053+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:40.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:40.201+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:40.312 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:40.201+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:40.510 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:40 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:40.325+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:40.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.4.service' 2024-04-24T23:49:40.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:40.574 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-24T23:49:40.574 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-24T23:49:40.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.5 2024-04-24T23:49:40.894 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 systemd[1]: Stopping Ceph osd.5 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:41.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:40 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:40.879+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:41.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:40 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:40.653+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:41 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:41.082+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:41 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:41.082+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:40.894+0000 7f28eaba3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:40.894+0000 7f28eaba3700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:40 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:40.894+0000 7f28eaba3700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:41.236 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:41 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:41.236+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:41.237 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:41 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:41.236+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:41.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:41 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:41.288+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:41.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:41 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:41.058+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:41.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:41 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:41.058+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:41.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:41 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:41.250+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:42.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:41.831+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:42.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:41.698+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:42.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:42 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:42.282+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:42.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:42 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:42.061+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:42.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:42 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:42.061+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:42.561 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:42 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:42.240+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:42.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:42 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:42.115+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:42.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:42 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:42.115+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:42.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:42 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:42.269+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:42.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:42 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:42.269+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:43.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:42 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:42.781+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:43.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:42 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:42.702+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:43.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:43 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:43.255+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:43.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:43 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:43.029+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:43.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:43 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:43.029+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:43.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:43 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:43.029+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:43.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:43.196+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:43.709 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:43.154+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:43.709 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:43.154+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:43.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:43.248+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:43.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:43.248+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:43.709 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:43 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:43.248+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:17.225552+0000 front 2024-04-24T23:49:17.225533+0000 (oldest deadline 2024-04-24T23:49:43.125633+0000) 2024-04-24T23:49:44.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:43.820+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:44.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:43 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:43.690+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:44.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:44.237+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:44.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:44.025+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:44.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:44.025+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:44.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:44 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:44.025+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:44.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:44.151+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:44.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:44.151+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:44.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:44.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:44.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:44.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:44.562 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:44.256+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:44.563 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:44.256+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:44.563 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:44 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:44.256+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:17.225552+0000 front 2024-04-24T23:49:17.225533+0000 (oldest deadline 2024-04-24T23:49:43.125633+0000) 2024-04-24T23:49:45.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:44.836+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:45.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:44.656+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:45.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:44 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:44.656+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:45.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:45 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:45.243+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:45.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:45 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:45.033+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:45.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:45 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:45.033+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:45.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:45 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:45.033+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:45.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:45.201+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:45.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:45.201+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:45.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:45.201+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:45.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:45.196+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:45.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:45.196+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:45.563 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:45.280+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:00.823415+0000 front 2024-04-24T23:49:00.823336+0000 (oldest deadline 2024-04-24T23:49:26.723473+0000) 2024-04-24T23:49:45.563 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:45.280+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:11.325155+0000 front 2024-04-24T23:49:11.325215+0000 (oldest deadline 2024-04-24T23:49:37.225225+0000) 2024-04-24T23:49:45.563 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:45 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5[44658]: 2024-04-24T23:49:45.280+0000 7f28e7197700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:17.225552+0000 front 2024-04-24T23:49:17.225533+0000 (oldest deadline 2024-04-24T23:49:43.125633+0000) 2024-04-24T23:49:46.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:45.843+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:46.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:45.843+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:46.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:45.648+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:46.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:45 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:45.648+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:46.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:46 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:46.242+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:46.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:46 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:46.066+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:46.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:46 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:46.066+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:46.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:46 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:46.066+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:46.522 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:46 smithi195 podman[67505]: 2024-04-24 23:49:46.247864435 +0000 UTC m=+5.476431228 container died 928480ed146ecc974e11f8dcedf14012c93644d229f49944db6f444819f0f0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-24T23:49:46.522 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:46 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:46.231+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:46.522 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:46 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:46.231+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:46.523 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:46 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:46.231+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:46.523 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:46 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:46.231+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:46.523 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:46 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:46.231+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:46.812 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:46 smithi195 podman[67505]: 2024-04-24 23:49:46.621373843 +0000 UTC m=+5.849940634 container cleanup 928480ed146ecc974e11f8dcedf14012c93644d229f49944db6f444819f0f0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:49:46.812 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:46 smithi195 bash[67505]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5 2024-04-24T23:49:47.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:46.817+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:47.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:46.817+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:47.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:46.655+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:47.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:46.655+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:47.217 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:46 smithi195 podman[67522]: 2024-04-24 23:49:46.92293972 +0000 UTC m=+0.668741019 container remove 928480ed146ecc974e11f8dcedf14012c93644d229f49944db6f444819f0f0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:47.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:47.202+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:47.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:47.202+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:47.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.023+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:47.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.023+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:47.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.023+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:47.542 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.145233441 +0000 UTC m=+0.018411864 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:47.542 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.277277444 +0000 UTC m=+0.150455867 container create 6aa7d4dd93d998742b540328209e59a5464533ef672b74e2bde973926421d67d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-24T23:49:47.542 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.54332396 +0000 UTC m=+0.416502383 container init 6aa7d4dd93d998742b540328209e59a5464533ef672b74e2bde973926421d67d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-24T23:49:47.543 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:47.217+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:47.543 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:47.217+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:47.543 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:47.224+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:47.543 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:47.224+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:47.543 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:47 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:47.224+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:47.812 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.546511653 +0000 UTC m=+0.419690075 container start 6aa7d4dd93d998742b540328209e59a5464533ef672b74e2bde973926421d67d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-24T23:49:47.812 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.643437215 +0000 UTC m=+0.516615658 container attach 6aa7d4dd93d998742b540328209e59a5464533ef672b74e2bde973926421d67d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-24T23:49:48.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:47.786+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:48.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:47.786+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:48.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:47.634+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:48.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:47 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:47.634+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:48.128 INFO:journalctl@ceph.osd.5.smithi195.stdout:Apr 24 23:49:47 smithi195 podman[67612]: 2024-04-24 23:49:47.832167872 +0000 UTC m=+0.705346304 container died 6aa7d4dd93d998742b540328209e59a5464533ef672b74e2bde973926421d67d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:48.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.974+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:48.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.974+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:48.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:47 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:47.974+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:48.438 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:48 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:48.173+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:48.438 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:48 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:48.173+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:48.438 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:48 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:48.187+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:48.439 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:48 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:48.187+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:48.439 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:48 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:48.187+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:48.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.5.service' 2024-04-24T23:49:48.500 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:48.500 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-24T23:49:48.500 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-24T23:49:48.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.6 2024-04-24T23:49:48.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:48 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:48.233+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:48.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:48 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:48.233+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:48.792 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 systemd[1]: Stopping Ceph osd.6 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:48.792 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:48.638+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:48.793 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:48.638+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:49.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:48.792+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:49.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:48.792+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:49.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:48.804+0000 7fb76beab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:49.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:48.804+0000 7fb76beab700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-24T23:49:49.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:48 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:48.804+0000 7fb76beab700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:49.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:48 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:48.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:49.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:48 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:48.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:49.232 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:48 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:48.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:49.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:49 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:49.214+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:49.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:49 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:49.214+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:49.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:49 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:49.161+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:49.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:49 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:49.161+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:49.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:49 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:49.161+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:49.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:49 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:49.233+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:49.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:49 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:49.233+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:50.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:49.749+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:50.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:49.749+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:50.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:49.678+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:50.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:49.678+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:50.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:49 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:49.988+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:50.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:49 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:49.988+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:50.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:49 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:49.988+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:50.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:50.220+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:50.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:50.220+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:50.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:50.220+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:50.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:50 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:50.224+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:50.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:50 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:50.224+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:50.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:50 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:50.191+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:50.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:50 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:50.191+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:50.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:50 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:50.191+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:51.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:50 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:50.759+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:51.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:50 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:50.759+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:51.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:50 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:50.663+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:51.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:50 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:50.663+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:51.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:51 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:51.218+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:51.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:51 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:51.218+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:51.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:51 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:51.218+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:51.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:50.994+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:51.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:50.994+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:51.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:50.994+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:51.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:50 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:50.994+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:51.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:51 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:51.274+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:51.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:51 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:51.274+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:51.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:51 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:51.163+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:51.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:51 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:51.163+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:51.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:51 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:51.163+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:52.127 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:51 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:51.737+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:52.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:51 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:51.737+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:52.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:51 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:51.737+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:25.610436+0000 front 2024-04-24T23:49:25.610466+0000 (oldest deadline 2024-04-24T23:49:51.510569+0000) 2024-04-24T23:49:52.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:51 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:51.698+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:52.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:51 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:51.698+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:52.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:52.207+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:52.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:52.207+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:52.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:52.207+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:52.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.013+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:52.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.013+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:52.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.013+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:52.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.014+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:52.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:52 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:52.262+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:52.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:52 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:52.262+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:52.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:52 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:52.143+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:52.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:52 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:52.143+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:52.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:52 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:52.143+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:53.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:52.660+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:53.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:52.660+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:53.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:52.694+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:53.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:52.694+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:53.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:52 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:52.694+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:25.610436+0000 front 2024-04-24T23:49:25.610466+0000 (oldest deadline 2024-04-24T23:49:51.510569+0000) 2024-04-24T23:49:53.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:53 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:53.222+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:53.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:53 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:53.222+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:53.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:53 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:53.222+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:53.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:53.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:53.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:53.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:52 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:52.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:53.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:53 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:53.264+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:53.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:53 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:53.264+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:53.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:53 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:53.166+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:53.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:53 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:53.166+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:53.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:53 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:53.166+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:54.028 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:53.715+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.208610+0000 front 2024-04-24T23:49:12.208518+0000 (oldest deadline 2024-04-24T23:49:36.908437+0000) 2024-04-24T23:49:54.028 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:53.715+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.310018+0000 front 2024-04-24T23:49:20.310042+0000 (oldest deadline 2024-04-24T23:49:45.610123+0000) 2024-04-24T23:49:54.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6[55144]: 2024-04-24T23:49:53.715+0000 7fb76849f700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:25.610436+0000 front 2024-04-24T23:49:25.610466+0000 (oldest deadline 2024-04-24T23:49:51.510569+0000) 2024-04-24T23:49:54.029 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:53.697+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:54.029 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:53.697+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:54.029 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:53 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:53.697+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:54.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:54.253+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:54.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:54.253+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:54.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:54.253+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:54.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:54.021+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:54.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:54.021+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:54.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:54.021+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:54.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:54 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:54.021+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:54.248+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:54.248+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:54.248+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:54.215+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:54.215+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:54.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:54 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:54.215+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:54.596 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:54 smithi107 podman[88402]: 2024-04-24 23:49:54.315168305 +0000 UTC m=+5.625934484 container died 80164eb0940150af108fb480cf8589a5ace34b624ad4e7d4ad1d6fb6a72a5287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-24T23:49:54.878 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:54 smithi107 podman[88402]: 2024-04-24 23:49:54.78823793 +0000 UTC m=+6.099004109 container cleanup 80164eb0940150af108fb480cf8589a5ace34b624ad4e7d4ad1d6fb6a72a5287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-24T23:49:54.878 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:54 smithi107 bash[88402]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6 2024-04-24T23:49:54.878 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:54 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:54.669+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:54.878 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:54 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:54.669+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:54.878 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:54 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:54.669+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:55.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:55.230+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:55.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:55.230+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:55.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:55.230+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:55.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:55.043+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:55.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:55.043+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:55.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:55.043+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:55.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:55 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:55.043+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:55.377 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88420]: 2024-04-24 23:49:55.089860255 +0000 UTC m=+0.777143084 container remove 80164eb0940150af108fb480cf8589a5ace34b624ad4e7d4ad1d6fb6a72a5287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:55.232+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:55.232+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:55.232+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:55.183+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:55.183+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:55.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:55 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:55.183+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:55.711 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.344525753 +0000 UTC m=+0.018279566 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:49:55.711 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.474026996 +0000 UTC m=+0.147780806 container create 2ea7fb4068c8e839e1c4771e094c79c1c0aeda0f06996911c8021a6cad9f7267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T23:49:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.685209422 +0000 UTC m=+0.358963229 container init 2ea7fb4068c8e839e1c4771e094c79c1c0aeda0f06996911c8021a6cad9f7267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-24T23:49:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.688899801 +0000 UTC m=+0.362653606 container start 2ea7fb4068c8e839e1c4771e094c79c1c0aeda0f06996911c8021a6cad9f7267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-24T23:49:55.977 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:55 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:55.711+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:55.978 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:55 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:55.711+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:55.978 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:55 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:55.711+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:55.978 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.785322541 +0000 UTC m=+0.459076350 container attach 2ea7fb4068c8e839e1c4771e094c79c1c0aeda0f06996911c8021a6cad9f7267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-24T23:49:56.279 INFO:journalctl@ceph.osd.6.smithi107.stdout:Apr 24 23:49:55 smithi107 podman[88520]: 2024-04-24 23:49:55.978850243 +0000 UTC m=+0.652604048 container died 2ea7fb4068c8e839e1c4771e094c79c1c0aeda0f06996911c8021a6cad9f7267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-24T23:49:56.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:56.204+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:56.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:56.204+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:56.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:56.204+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:56.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:56.020+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:56.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:56.020+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:56.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:56.020+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:56.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:56.020+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:56.266+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:56.266+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:56.266+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:56.230+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:56.230+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:56.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:56 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:56.230+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:56.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.6.service' 2024-04-24T23:49:56.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:49:56.762 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-24T23:49:56.762 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-24T23:49:56.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.7 2024-04-24T23:49:57.078 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.024+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:57.078 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.024+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:57.079 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.024+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:57.079 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.024+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:57.079 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:56 smithi170 systemd[1]: Stopping Ceph osd.7 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:49:57.079 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.078+0000 7fde22a82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:49:57.079 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.078+0000 7fde22a82700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-24T23:49:57.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:56 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:56.756+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:57.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:56 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:56.756+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:57.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:56 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:56.756+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:57.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.078+0000 7fde22a82700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:49:57.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.162+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:57.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.162+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:57.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:57.162+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:57.570 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:57.247+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:57.570 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:57.247+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:57.570 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:57.247+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:57.570 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:57.251+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:57.570 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:57.251+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:57.571 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:57 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:57.251+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:58.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:57 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:57.793+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:58.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:57 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:57.793+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:58.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:57 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:57.793+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:58.181+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:58.181+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:58.181+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:58.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:57 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:57.983+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:58.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:58.215+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:58.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:58.215+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:58.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:58.215+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:58.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:58.249+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:58.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:58.249+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:58.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:58 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:58.249+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:59.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:58 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:58.785+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:49:59.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:58 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:58.785+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:49:59.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:58 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:58.785+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:49:59.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:59.183+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:49:59.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:59.183+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:49:59.335 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:49:59.183+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:49:59.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:58.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:49:59.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:58.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:49:59.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:58.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:49:59.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:58 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:58.941+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:49:59.463 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:59.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:49:59.463 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:59.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:49:59.463 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:49:59.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:49:59.464 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:59.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:49:59.464 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:59.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:49:59.464 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:59.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:49:59.464 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:49:59 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:49:59.181+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:00.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:59 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:59.766+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:00.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:59 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:59.766+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:00.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:49:59 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:49:59.766+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:00.179 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:59.924+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:00.180 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:59.924+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:00.180 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:59.924+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:00.180 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:59.924+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:00.180 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:49:59 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:49:59.924+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:00.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:00.340+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:00.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:00.340+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:00.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:00.340+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:00.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:00.213+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:00.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:00.213+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:00.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:00.213+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:00.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:00 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:00.213+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:00.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:00.179+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:50:00.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:00.179+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:50:00.585 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:00.179+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:50:01.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:00.778+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:01.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:00.778+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:01.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:00 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:00.778+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:01.210 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:00.951+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:01.211 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:00.951+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:01.211 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:00.951+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:01.211 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:00.951+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:01.211 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:00 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:00.951+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:01.214 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:01.210+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.980024+0000 front 2024-04-24T23:49:12.980155+0000 (oldest deadline 2024-04-24T23:49:35.880174+0000) 2024-04-24T23:50:01.214 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:01.210+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:20.581431+0000 front 2024-04-24T23:49:20.581483+0000 (oldest deadline 2024-04-24T23:49:46.481465+0000) 2024-04-24T23:50:01.214 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7[50762]: 2024-04-24T23:50:01.210+0000 7fde1b06e700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:26.482162+0000 front 2024-04-24T23:49:26.482122+0000 (oldest deadline 2024-04-24T23:49:49.982093+0000) 2024-04-24T23:50:01.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:01.175+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:01.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:01.175+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:01.175+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:01.175+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:01.325+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:01.325+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:01.325+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:01.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:01 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:01.325+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:02.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:01 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:01.807+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:02.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:01 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:01.807+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:02.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:01 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:01.807+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:02.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:01 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:01.807+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:02.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:01.966+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:02.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:01.966+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:02.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:01.966+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:02.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:01.966+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:02.336 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:01.966+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:02.129+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:02.129+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:02.129+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:02.129+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:02.333+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:02.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:02.333+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:02.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:02.333+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:02.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:02 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:02.333+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:02.696 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:02 smithi170 podman[66845]: 2024-04-24 23:50:02.435376723 +0000 UTC m=+5.476886035 container died e8cdd1c5d58667dea75cf8a6ed060c744f63342a1550ee0b3b1b2c897028da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-24T23:50:03.085 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:02 smithi170 podman[66845]: 2024-04-24 23:50:02.796137252 +0000 UTC m=+5.837646552 container cleanup e8cdd1c5d58667dea75cf8a6ed060c744f63342a1550ee0b3b1b2c897028da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T23:50:03.085 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:02 smithi170 bash[66845]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7 2024-04-24T23:50:03.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:02.925+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:03.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:02.925+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:03.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:02.925+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:03.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:02.925+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:03.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:02 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:02.925+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:03.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:02.834+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:03.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:02.834+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:03.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:02.834+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:03.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:02 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:02.834+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:03.472 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66863]: 2024-04-24 23:50:03.115017782 +0000 UTC m=+0.677692437 container remove e8cdd1c5d58667dea75cf8a6ed060c744f63342a1550ee0b3b1b2c897028da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T23:50:03.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:03.153+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:03.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:03.153+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:03.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:03.153+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:03.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:03.153+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:03.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:03.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:03.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:03.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:03.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:03.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:03.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:03 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:03.291+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:03.725 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66953]: 2024-04-24 23:50:03.473278141 +0000 UTC m=+0.113863376 container create 18e6e5bf4f7d1c67f6ca62b81ea558367953d7d959616e10565350d97fa54d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T23:50:03.725 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66953]: 2024-04-24 23:50:03.377529932 +0000 UTC m=+0.018115168 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66953]: 2024-04-24 23:50:03.725641674 +0000 UTC m=+0.366226914 container init 18e6e5bf4f7d1c67f6ca62b81ea558367953d7d959616e10565350d97fa54d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66953]: 2024-04-24 23:50:03.729561721 +0000 UTC m=+0.370146962 container start 18e6e5bf4f7d1c67f6ca62b81ea558367953d7d959616e10565350d97fa54d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:03 smithi170 podman[66953]: 2024-04-24 23:50:03.800712164 +0000 UTC m=+0.441297413 container attach 18e6e5bf4f7d1c67f6ca62b81ea558367953d7d959616e10565350d97fa54d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:03.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:03.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:03.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:03.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:04.012 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:03 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:03.877+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:04.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:03 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:03.842+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:04.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:03 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:03.842+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:04.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:03 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:03.842+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:04.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:03 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:03.842+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:04.281 INFO:journalctl@ceph.osd.7.smithi170.stdout:Apr 24 23:50:04 smithi170 podman[66953]: 2024-04-24 23:50:04.013015633 +0000 UTC m=+0.653600871 container died 18e6e5bf4f7d1c67f6ca62b81ea558367953d7d959616e10565350d97fa54d7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:04.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:04.147+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:04.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:04.148+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:04.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:04.148+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:04.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:04.148+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:04.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:04.316+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:04.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:04.316+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:04.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:04.316+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:04.563 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:04 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:04.316+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:04.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.7.service' 2024-04-24T23:50:04.676 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:04.677 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-24T23:50:04.677 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-24T23:50:04.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.8 2024-04-24T23:50:05.035 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:04 smithi195 systemd[1]: Stopping Ceph osd.8 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:05.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:04.872+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:05.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:04.872+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:05.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:04.872+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:05.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:04 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:04.872+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:05.220 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:04.897+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:05.221 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:04.897+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:05.221 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:04.897+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:05.221 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:04.897+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:05.221 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:04 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:04.897+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.036+0000 7f2ab7a52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.036+0000 7f2ab7a52700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.036+0000 7f2ab7a52700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.111+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.111+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.111+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:05.312 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:05.111+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:05.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:05.345+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:05.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:05.345+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:05.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:05.345+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:05.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:05 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:05.345+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:06.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:05.905+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:06.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:05.905+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:06.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:05.905+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:06.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:05 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:05.905+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:06.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:05.859+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:06.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:05.859+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:06.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:05.859+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:06.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:05.859+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:06.335 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:05.859+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:06.139+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:06.139+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:06.139+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:06.139+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:06.380+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:06.380+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:06.380+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:06.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:06 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:06.380+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:07.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:06 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:06.834+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:07.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:06 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:06.834+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:07.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:06 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:06.834+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:07.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:06 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:06.834+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:07.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:06 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:06.834+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:07.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:06.925+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:07.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:06.925+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:07.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:06.925+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:07.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:06.925+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:07.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:06 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:06.925+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:07.133+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:07.133+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:07.133+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:07.133+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:07.337+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:07.337+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:07.337+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:07.337+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:07.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:07 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:07.337+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:08.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:07 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:07.814+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:08.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:07 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:07.814+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:08.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:07 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:07.814+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:08.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:07 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:07.814+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:08.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:07 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:07.814+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:08.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:07.974+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:08.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:07.974+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:08.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:07.974+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:08.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:07.974+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:08.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:07 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:07.974+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:08.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:08.098+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:08.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:08.098+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:08.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:08.098+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:08.562 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:08.098+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:08.565 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:08.327+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:08.565 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:08.327+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:08.566 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:08.327+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:08.566 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:08.327+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:08.566 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:08 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:08.327+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:09.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:08.803+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:09.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:08.803+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:09.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:08.803+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:09.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:08.803+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:09.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:08 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:08.803+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:09.356 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:09.063+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:09.836312+0000 front 2024-04-24T23:49:09.836219+0000 (oldest deadline 2024-04-24T23:49:35.736279+0000) 2024-04-24T23:50:09.356 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:09.063+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.237320+0000 front 2024-04-24T23:49:19.237552+0000 (oldest deadline 2024-04-24T23:49:43.937495+0000) 2024-04-24T23:50:09.356 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:09.063+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.238761+0000 front 2024-04-24T23:49:29.238799+0000 (oldest deadline 2024-04-24T23:49:53.338629+0000) 2024-04-24T23:50:09.356 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:09.063+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:35.639822+0000 front 2024-04-24T23:49:35.639850+0000 (oldest deadline 2024-04-24T23:49:58.539553+0000) 2024-04-24T23:50:09.356 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8[51635]: 2024-04-24T23:50:09.063+0000 7f2ab003e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:43.140904+0000 front 2024-04-24T23:49:43.140936+0000 (oldest deadline 2024-04-24T23:50:08.440932+0000) 2024-04-24T23:50:09.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:08.929+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:09.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:08.929+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:09.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:08.929+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:09.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:08.929+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:09.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:08 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:08.929+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:09.811 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:09.356+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:09.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:09.356+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:09.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:09.356+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:09.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:09.356+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:09.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:09.356+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:10.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:10.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:10.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:10.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:10.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:10.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:09 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:09.760+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:10.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:09 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:09.970+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:10.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:09 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:09.970+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:10.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:09 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:09.970+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:10.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:09 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:09.970+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:10.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:09 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:09.970+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:10.466 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:10 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:10.392+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:10.466 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:10 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:10.392+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:10.466 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:10 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:10.392+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:10.466 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:10 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:10.392+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:10.466 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:10 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:10.392+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:10.726 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:10 smithi195 podman[67732]: 2024-04-24 23:50:10.466827317 +0000 UTC m=+5.594862330 container died 6192bbb307dfda1823d7cf45d151a09c909a4208d686bd877f8936fdfcb7c4ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-24T23:50:11.061 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:10 smithi195 podman[67732]: 2024-04-24 23:50:10.826009803 +0000 UTC m=+5.954044806 container cleanup 6192bbb307dfda1823d7cf45d151a09c909a4208d686bd877f8936fdfcb7c4ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:11.062 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:10 smithi195 bash[67732]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8 2024-04-24T23:50:11.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:11.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:11.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:11.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:11.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:11.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:10 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:10.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:11.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.010+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:11.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.010+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:11.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.010+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:11.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.010+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:11.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.010+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:11.483 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67749]: 2024-04-24 23:50:11.160818731 +0000 UTC m=+0.687675537 container remove 6192bbb307dfda1823d7cf45d151a09c909a4208d686bd877f8936fdfcb7c4ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-24T23:50:11.484 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:11 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:11.366+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:11.484 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:11 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:11.366+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:11.484 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:11 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:11.366+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:11.484 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:11 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:11.366+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:11.484 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:11 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:11.366+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:11.797 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67840]: 2024-04-24 23:50:11.387493655 +0000 UTC m=+0.018341894 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:50:11.797 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67840]: 2024-04-24 23:50:11.495193719 +0000 UTC m=+0.126041961 container create 19bde961c8c50b3d1d585d1e39feed42b34b8e1d0a80894804b4956fe8954a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-24T23:50:11.797 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67840]: 2024-04-24 23:50:11.723312049 +0000 UTC m=+0.354160290 container init 19bde961c8c50b3d1d585d1e39feed42b34b8e1d0a80894804b4956fe8954a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T23:50:11.797 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67840]: 2024-04-24 23:50:11.72623201 +0000 UTC m=+0.357080256 container start 19bde961c8c50b3d1d585d1e39feed42b34b8e1d0a80894804b4956fe8954a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:11.798 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:11 smithi195 podman[67840]: 2024-04-24 23:50:11.798199732 +0000 UTC m=+0.429047986 container attach 19bde961c8c50b3d1d585d1e39feed42b34b8e1d0a80894804b4956fe8954a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T23:50:12.062 INFO:journalctl@ceph.osd.8.smithi195.stdout:Apr 24 23:50:12 smithi195 podman[67840]: 2024-04-24 23:50:12.013968844 +0000 UTC m=+0.644817088 container died 19bde961c8c50b3d1d585d1e39feed42b34b8e1d0a80894804b4956fe8954a34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T23:50:12.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:12.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:12.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:12.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:12.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:12.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:11 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:11.707+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:12.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.979+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:12.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.979+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:12.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.979+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:12.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.979+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:12.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:11 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:11.979+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:12.561 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:12.371+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:12.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:12.371+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:12.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:12.371+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:12.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:12.371+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:12.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:12 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:12.371+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:12.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.8.service' 2024-04-24T23:50:12.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:12.698 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-24T23:50:12.698 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-24T23:50:12.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.9 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 systemd[1]: Stopping Ceph osd.9 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.972+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.972+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.972+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.972+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.972+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:12.990 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.990+0000 7f4513f6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:12.991 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.990+0000 7f4513f6f700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-24T23:50:12.991 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:12 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:12.990+0000 7f4513f6f700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:13.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:12 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:12.721+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:13.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:13.408+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:13.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:13.408+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:13.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:13.408+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:13.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:13.408+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:13.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:13.408+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:14.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:14.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:14.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:14.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:14.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:14.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:13 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:13.771+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:14.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:13 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:13.958+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:14.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:13 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:13.958+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:14.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:13 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:13.958+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:14.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:13 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:13.958+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:14.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:13 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:13.958+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:14.811 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:14 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:14.417+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:14.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:14 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:14.417+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:14.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:14 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:14.417+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:14.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:14 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:14.417+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:14.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:14 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:14.417+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:15.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:15.250 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:15.250 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:15.250 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:15.250 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:15.251 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:14 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:14.768+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:15.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:15.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:15.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:15.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:15.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:15.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:14 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:14.916+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.986614+0000 front 2024-04-24T23:49:51.986656+0000 (oldest deadline 2024-04-24T23:50:14.886762+0000) 2024-04-24T23:50:15.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:15 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:15.454+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:15.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:15 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:15.454+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:15.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:15 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:15.454+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:15.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:15 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:15.454+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:15.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:15 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:15.454+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:16.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:16.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:15 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:15.776+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:16.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:16.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:16.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:16.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:16.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:16.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:15 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:15.888+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.986614+0000 front 2024-04-24T23:49:51.986656+0000 (oldest deadline 2024-04-24T23:50:14.886762+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:16.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:16 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:16.494+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:17.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:17.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:16 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:16.731+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:17.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:17.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:17.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:17.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:17.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:17.379 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:16 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:16.898+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.986614+0000 front 2024-04-24T23:49:51.986656+0000 (oldest deadline 2024-04-24T23:50:14.886762+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:17.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:17.451+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:18.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:18.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:18.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:18.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:18.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:18.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:18.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:17 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:17.715+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:18.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:13.282176+0000 front 2024-04-24T23:49:13.282142+0000 (oldest deadline 2024-04-24T23:49:37.381985+0000) 2024-04-24T23:50:18.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:19.083020+0000 front 2024-04-24T23:49:19.083116+0000 (oldest deadline 2024-04-24T23:49:44.383037+0000) 2024-04-24T23:50:18.377 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:29.084126+0000 front 2024-04-24T23:49:29.084030+0000 (oldest deadline 2024-04-24T23:49:53.184213+0000) 2024-04-24T23:50:18.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.685442+0000 front 2024-04-24T23:49:36.685559+0000 (oldest deadline 2024-04-24T23:50:00.785199+0000) 2024-04-24T23:50:18.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:40.785705+0000 front 2024-04-24T23:49:40.785669+0000 (oldest deadline 2024-04-24T23:50:06.685702+0000) 2024-04-24T23:50:18.378 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:17 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9[62032]: 2024-04-24T23:50:17.941+0000 7f4510563700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.986614+0000 front 2024-04-24T23:49:51.986656+0000 (oldest deadline 2024-04-24T23:50:14.886762+0000) 2024-04-24T23:50:18.705 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:18 smithi107 podman[88627]: 2024-04-24 23:50:18.397719321 +0000 UTC m=+5.513302738 container died 1b795252d77c189e97763d5eda5118562f2f60b3924e6322b4a73b30930464c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-24T23:50:18.705 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:18 smithi107 podman[88627]: 2024-04-24 23:50:18.706700313 +0000 UTC m=+5.822283730 container cleanup 1b795252d77c189e97763d5eda5118562f2f60b3924e6322b4a73b30930464c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:18.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:18 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:18.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:19.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:19.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:19.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:19.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:19.087 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:19.087 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:19.087 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:18 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:18.683+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:19.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:18 smithi107 bash[88627]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9 2024-04-24T23:50:19.127 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:18 smithi107 podman[88644]: 2024-04-24 23:50:18.891346894 +0000 UTC m=+0.495603738 container remove 1b795252d77c189e97763d5eda5118562f2f60b3924e6322b4a73b30930464c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T23:50:19.627 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.233873946 +0000 UTC m=+0.018072682 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:50:19.627 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.341983279 +0000 UTC m=+0.126182021 container create 072a9352ce75281a1b8727dece56d98b9a92e7cb137514735d4022c5608714c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-24T23:50:19.628 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.569659625 +0000 UTC m=+0.353858361 container init 072a9352ce75281a1b8727dece56d98b9a92e7cb137514735d4022c5608714c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T23:50:19.628 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.572837619 +0000 UTC m=+0.357036358 container start 072a9352ce75281a1b8727dece56d98b9a92e7cb137514735d4022c5608714c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:19.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:19 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:19.421+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:20.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:20.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:19 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:19.662+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:20.128 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.655406524 +0000 UTC m=+0.439605264 container attach 072a9352ce75281a1b8727dece56d98b9a92e7cb137514735d4022c5608714c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:50:20.132 INFO:journalctl@ceph.osd.9.smithi107.stdout:Apr 24 23:50:19 smithi107 podman[88732]: 2024-04-24 23:50:19.864329604 +0000 UTC m=+0.648528342 container died 072a9352ce75281a1b8727dece56d98b9a92e7cb137514735d4022c5608714c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-24T23:50:20.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.9.service' 2024-04-24T23:50:20.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:20.490 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-24T23:50:20.490 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-24T23:50:20.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.10 2024-04-24T23:50:20.807 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 systemd[1]: Stopping Ceph osd.10 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:20.807 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:20.808 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.689+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:20.809 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.806+0000 7f42b2d1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:20.809 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.806+0000 7f42b2d1e700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-24T23:50:20.809 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:20 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:20.806+0000 7f42b2d1e700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:50:20.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:20.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:20.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:20.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:20.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:20.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:20 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:20.419+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:21.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:21.437+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:22.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:22.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:22.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:22.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:22.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:22.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:22.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:21 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:21.640+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:22.719 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:22.720 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:22.720 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:22.720 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:22.720 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:22.720 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:22 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:22.449+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:23.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:23.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:23.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:23.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:23.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:23.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:23.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:22 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:22.611+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:23.562 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:23 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:23.497+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:24.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:24.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:24.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:24.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:24.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:24.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:24.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:23 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:23.644+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:24.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:24 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:24.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:25.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:25.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:25.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:25.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:25.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:25.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:25.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:24 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:24.687+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:25.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:25 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:25.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:26.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.246514+0000 front 2024-04-24T23:49:04.246483+0000 (oldest deadline 2024-04-24T23:49:28.346485+0000) 2024-04-24T23:50:26.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:10.547905+0000 front 2024-04-24T23:49:10.548072+0000 (oldest deadline 2024-04-24T23:49:35.848048+0000) 2024-04-24T23:50:26.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:15.848585+0000 front 2024-04-24T23:49:15.848691+0000 (oldest deadline 2024-04-24T23:49:42.249100+0000) 2024-04-24T23:50:26.085 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6814 osd.3 since back 2024-04-24T23:49:27.550470+0000 front 2024-04-24T23:49:27.550377+0000 (oldest deadline 2024-04-24T23:49:50.450250+0000) 2024-04-24T23:50:26.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:33.851856+0000 front 2024-04-24T23:49:33.851895+0000 (oldest deadline 2024-04-24T23:49:59.751963+0000) 2024-04-24T23:50:26.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:44.452915+0000 front 2024-04-24T23:49:44.452952+0000 (oldest deadline 2024-04-24T23:50:09.152711+0000) 2024-04-24T23:50:26.086 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:25 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10[57659]: 2024-04-24T23:50:25.659+0000 7f42af312700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:53.253577+0000 front 2024-04-24T23:49:53.253722+0000 (oldest deadline 2024-04-24T23:50:14.953523+0000) 2024-04-24T23:50:26.507 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:26 smithi170 podman[67069]: 2024-04-24 23:50:26.241834111 +0000 UTC m=+5.550753187 container died 4aad0317fbf6790eb032cbd84e490c0a9667e7048bedabc5113005fc1678b758 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:26.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:26 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:26.485+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:26.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:26 smithi170 podman[67069]: 2024-04-24 23:50:26.598220091 +0000 UTC m=+5.907139155 container cleanup 4aad0317fbf6790eb032cbd84e490c0a9667e7048bedabc5113005fc1678b758 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:26.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:26 smithi170 bash[67069]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10 2024-04-24T23:50:27.230 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:26 smithi170 podman[67086]: 2024-04-24 23:50:26.85759511 +0000 UTC m=+0.616391926 container remove 4aad0317fbf6790eb032cbd84e490c0a9667e7048bedabc5113005fc1678b758 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:27.551 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.134467995 +0000 UTC m=+0.018211865 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:50:27.552 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.250355212 +0000 UTC m=+0.134099067 container create fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-24T23:50:27.552 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.46935484 +0000 UTC m=+0.353098692 container init fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:27.552 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.472372388 +0000 UTC m=+0.356116246 container start fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-24T23:50:27.552 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.552618302 +0000 UTC m=+0.436362157 container attach fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:27.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:27 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:27.488+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:27.835 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:27 smithi170 podman[67173]: 2024-04-24 23:50:27.754623754 +0000 UTC m=+0.638367610 container died fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:28.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.10.service' 2024-04-24T23:50:28.288 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:28 smithi170 podman[67199]: 2024-04-24 23:50:28.247426496 +0000 UTC m=+0.487439268 container remove fdf3afbfbc666ea2ca8743df5a3878c8e17a3373eb863ab4e6eb3a362470db11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:28.288 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:28 smithi170 systemd[1]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.10.service: Deactivated successfully. 2024-04-24T23:50:28.288 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:28 smithi170 systemd[1]: Stopped Ceph osd.10 for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:50:28.288 INFO:journalctl@ceph.osd.10.smithi170.stdout:Apr 24 23:50:28 smithi170 systemd[1]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.10.service: Consumed 20.587s CPU time. 2024-04-24T23:50:28.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:28.303 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-24T23:50:28.303 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-24T23:50:28.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.11 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 systemd[1]: Stopping Ceph osd.11 for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:28.596 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.479+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:29.061 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.595+0000 7f815d5ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:29.062 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.595+0000 7f815d5ff700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-24T23:50:29.062 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:28 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:28.595+0000 7f815d5ff700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T23:50:29.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:29.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:29 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:29.493+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:30.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:30.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:30.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:30.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:30.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:30.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:30.813 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:30 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:30.513+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:31.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:31 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:31.470+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:32.811 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:32.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:32 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:32.504+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-04-24T23:49:04.472144+0000 front 2024-04-24T23:49:04.471920+0000 (oldest deadline 2024-04-24T23:49:26.172060+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6804 osd.1 since back 2024-04-24T23:49:12.073306+0000 front 2024-04-24T23:49:12.073333+0000 (oldest deadline 2024-04-24T23:49:36.173451+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6804 osd.2 since back 2024-04-24T23:49:16.674406+0000 front 2024-04-24T23:49:16.674486+0000 (oldest deadline 2024-04-24T23:49:44.874950+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6812 osd.4 since back 2024-04-24T23:49:36.477830+0000 front 2024-04-24T23:49:36.477845+0000 (oldest deadline 2024-04-24T23:50:01.177643+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-04-24T23:49:41.177906+0000 front 2024-04-24T23:49:41.177925+0000 (oldest deadline 2024-04-24T23:50:07.078084+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.107:6822 osd.6 since back 2024-04-24T23:49:51.079560+0000 front 2024-04-24T23:49:51.079575+0000 (oldest deadline 2024-04-24T23:50:16.379367+0000) 2024-04-24T23:50:33.812 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:33 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11[58233]: 2024-04-24T23:50:33.483+0000 7f8159bf3700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.170:6820 osd.7 since back 2024-04-24T23:50:00.480322+0000 front 2024-04-24T23:50:00.480376+0000 (oldest deadline 2024-04-24T23:50:23.980197+0000) 2024-04-24T23:50:34.271 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 podman[67957]: 2024-04-24 23:50:34.013783168 +0000 UTC m=+5.520228470 container died 483beb518f19f6b3067ba28be08a56c7da051cf0151a41d5cc379985285c10a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-24T23:50:34.547 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 podman[67957]: 2024-04-24 23:50:34.362842237 +0000 UTC m=+5.869287576 container cleanup 483beb518f19f6b3067ba28be08a56c7da051cf0151a41d5cc379985285c10a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T23:50:34.547 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 bash[67957]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11 2024-04-24T23:50:34.811 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 podman[67990]: 2024-04-24 23:50:34.5474971 +0000 UTC m=+0.528307617 container remove 483beb518f19f6b3067ba28be08a56c7da051cf0151a41d5cc379985285c10a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-24T23:50:35.250 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 podman[68081]: 2024-04-24 23:50:34.877667557 +0000 UTC m=+0.017967562 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-24T23:50:35.250 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:34 smithi195 podman[68081]: 2024-04-24 23:50:34.981651188 +0000 UTC m=+0.121951180 container create 1d885c202f1c200dfa9928dfefc4bbf27ac5ae8f080ab88fb4209a51cc5f8b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-24T23:50:35.251 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:35 smithi195 podman[68081]: 2024-04-24 23:50:35.251075264 +0000 UTC m=+0.391375255 container init 1d885c202f1c200dfa9928dfefc4bbf27ac5ae8f080ab88fb4209a51cc5f8b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-24T23:50:35.543 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:35 smithi195 podman[68081]: 2024-04-24 23:50:35.254661287 +0000 UTC m=+0.394961279 container start 1d885c202f1c200dfa9928dfefc4bbf27ac5ae8f080ab88fb4209a51cc5f8b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T23:50:35.543 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:35 smithi195 podman[68081]: 2024-04-24 23:50:35.33456099 +0000 UTC m=+0.474860997 container attach 1d885c202f1c200dfa9928dfefc4bbf27ac5ae8f080ab88fb4209a51cc5f8b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T23:50:35.811 INFO:journalctl@ceph.osd.11.smithi195.stdout:Apr 24 23:50:35 smithi195 podman[68081]: 2024-04-24 23:50:35.544238686 +0000 UTC m=+0.684538676 container died 1d885c202f1c200dfa9928dfefc4bbf27ac5ae8f080ab88fb4209a51cc5f8b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-24T23:50:36.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@osd.11.service' 2024-04-24T23:50:36.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:36.147 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-24T23:50:36.147 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-24T23:50:36.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.a 2024-04-24T23:50:36.457 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:36 smithi107 systemd[1]: Stopping Ceph mds.a for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:36.877 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:36 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: 2024-04-24T23:50:36.456+0000 7fc5cc50a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:36.877 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:36 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a[68748]: 2024-04-24T23:50:36.456+0000 7fc5cc50a700 -1 mds.a *** got signal Terminated *** 2024-04-24T23:50:40.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:39 smithi107 podman[88854]: 2024-04-24 23:50:39.878825385 +0000 UTC m=+3.539546094 container died 19c771eaab50c0cc1287aa641c7f5b56fd333bc2fa3027a8946c4b26ab1889ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:40.498 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:40 smithi107 podman[88854]: 2024-04-24 23:50:40.289987567 +0000 UTC m=+3.950708273 container cleanup 19c771eaab50c0cc1287aa641c7f5b56fd333bc2fa3027a8946c4b26ab1889ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T23:50:40.498 INFO:journalctl@ceph.mds.a.smithi107.stdout:Apr 24 23:50:40 smithi107 bash[88854]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-a 2024-04-24T23:50:40.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.a.service' 2024-04-24T23:50:40.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:40.683 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-24T23:50:40.683 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-24T23:50:40.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.d 2024-04-24T23:50:41.127 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:40 smithi107 systemd[1]: Stopping Ceph mds.d for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:41.128 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d[67781]: 2024-04-24T23:50:41.000+0000 7f589d5da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:41.128 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:41 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d[67781]: 2024-04-24T23:50:41.000+0000 7f589d5da700 -1 mds.d *** got signal Terminated *** 2024-04-24T23:50:45.309 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:45 smithi107 podman[88976]: 2024-04-24 23:50:45.006855626 +0000 UTC m=+4.105527914 container died 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T23:50:45.627 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:45 smithi107 podman[88976]: 2024-04-24 23:50:45.401338595 +0000 UTC m=+4.500010870 container cleanup 0649073ac5a659486d94dd10061a17b08e56e79b1369e09f91fe704c257d7f45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T23:50:45.627 INFO:journalctl@ceph.mds.d.smithi107.stdout:Apr 24 23:50:45 smithi107 bash[88976]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-d 2024-04-24T23:50:45.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.d.service' 2024-04-24T23:50:45.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:45.813 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-24T23:50:45.813 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-24T23:50:45.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.g 2024-04-24T23:50:46.127 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:45 smithi107 systemd[1]: Stopping Ceph mds.g for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:46.627 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: 2024-04-24T23:50:46.156+0000 7f9dd9c28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:46.627 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:46 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g[68391]: 2024-04-24T23:50:46.156+0000 7f9dd9c28700 -1 mds.g *** got signal Terminated *** 2024-04-24T23:50:48.154 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:47 smithi107 podman[89099]: 2024-04-24 23:50:47.886038992 +0000 UTC m=+1.858044896 container died cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-24T23:50:48.513 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:48 smithi107 podman[89099]: 2024-04-24 23:50:48.237937824 +0000 UTC m=+2.209943724 container cleanup cf4a201e67e9199e22969da9863a89e9217899f41c2d7ed8de142287256ee790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-24T23:50:48.513 INFO:journalctl@ceph.mds.g.smithi107.stdout:Apr 24 23:50:48 smithi107 bash[89099]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-g 2024-04-24T23:50:48.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.g.service' 2024-04-24T23:50:48.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:48.695 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-24T23:50:48.696 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-24T23:50:48.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.j 2024-04-24T23:50:49.127 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:48 smithi107 systemd[1]: Stopping Ceph mds.j for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:49.127 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j[67393]: 2024-04-24T23:50:49.024+0000 7fde750f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:49.127 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:49 smithi107 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j[67393]: 2024-04-24T23:50:49.024+0000 7fde750f7700 -1 mds.j *** got signal Terminated *** 2024-04-24T23:50:50.540 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:50 smithi107 podman[89222]: 2024-04-24 23:50:50.264842197 +0000 UTC m=+1.350812990 container died 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T23:50:50.540 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:50 smithi107 podman[89222]: 2024-04-24 23:50:50.541633264 +0000 UTC m=+1.627604057 container cleanup 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T23:50:50.877 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:50 smithi107 bash[89222]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j 2024-04-24T23:50:50.877 INFO:journalctl@ceph.mds.j.smithi107.stdout:Apr 24 23:50:50 smithi107 podman[89236]: 2024-04-24 23:50:50.767705424 +0000 UTC m=+0.501795691 container remove 17ce9952d998db013a34af20de02c0c5c5e9fe86c1adfdb1bd650c7993558b83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:50:50.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.j.service' 2024-04-24T23:50:50.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:50.943 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-24T23:50:50.943 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-24T23:50:50.943 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.b 2024-04-24T23:50:51.239 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:50 smithi170 systemd[1]: Stopping Ceph mds.b for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:51.553 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:51 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: 2024-04-24T23:50:51.239+0000 7f7d047d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:51.553 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:51 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b[63066]: 2024-04-24T23:50:51.239+0000 7f7d047d1700 -1 mds.b *** got signal Terminated *** 2024-04-24T23:50:55.505 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:55 smithi170 podman[67280]: 2024-04-24 23:50:55.246982931 +0000 UTC m=+4.108146093 container died 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T23:50:55.806 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:55 smithi170 podman[67280]: 2024-04-24 23:50:55.58088376 +0000 UTC m=+4.442046912 container cleanup 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-24T23:50:55.806 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:55 smithi170 bash[67280]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b 2024-04-24T23:50:56.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.b.service' 2024-04-24T23:50:56.071 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:55 smithi170 podman[67303]: 2024-04-24 23:50:55.807097017 +0000 UTC m=+0.555035136 container remove 2322592b7fb6abcaddcd09301423ef2298da24be42cf8145cd4851348544a01d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:50:56.071 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:56 smithi170 systemd[1]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-24T23:50:56.071 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:56 smithi170 systemd[1]: Stopped Ceph mds.b for 16fa2ae8-0293-11ef-bc93-c7b262605968. 2024-04-24T23:50:56.071 INFO:journalctl@ceph.mds.b.smithi170.stdout:Apr 24 23:50:56 smithi170 systemd[1]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.b.service: Consumed 13.614s CPU time. 2024-04-24T23:50:56.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:50:56.085 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-24T23:50:56.085 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-24T23:50:56.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.e 2024-04-24T23:50:56.335 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:50:56 smithi170 systemd[1]: Stopping Ceph mds.e for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:50:56.835 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:50:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: 2024-04-24T23:50:56.442+0000 7ff05503c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:50:56.835 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:50:56 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e[62430]: 2024-04-24T23:50:56.442+0000 7ff05503c700 -1 mds.e *** got signal Terminated *** 2024-04-24T23:51:00.225 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:50:59 smithi170 podman[67409]: 2024-04-24 23:50:59.905284883 +0000 UTC m=+3.600070685 container died ea811e32727d137102c8f172e7ec87873c6b825943b2c5c5041be330c12ad73c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T23:51:00.225 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:51:00 smithi170 podman[67409]: 2024-04-24 23:51:00.2261391 +0000 UTC m=+3.920924903 container cleanup ea811e32727d137102c8f172e7ec87873c6b825943b2c5c5041be330c12ad73c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-24T23:51:00.585 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:51:00 smithi170 bash[67409]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e 2024-04-24T23:51:00.585 INFO:journalctl@ceph.mds.e.smithi170.stdout:Apr 24 23:51:00 smithi170 podman[67423]: 2024-04-24 23:51:00.427316757 +0000 UTC m=+0.519235301 container remove ea811e32727d137102c8f172e7ec87873c6b825943b2c5c5041be330c12ad73c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-24T23:51:00.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.e.service' 2024-04-24T23:51:00.691 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:00.691 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-24T23:51:00.691 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-24T23:51:00.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.h 2024-04-24T23:51:01.003 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:00 smithi170 systemd[1]: Stopping Ceph mds.h for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:01.335 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: 2024-04-24T23:51:01.004+0000 7fd083ac7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:01.335 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:01 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h[62742]: 2024-04-24T23:51:01.004+0000 7fd083ac7700 -1 mds.h *** got signal Terminated *** 2024-04-24T23:51:04.171 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:03 smithi170 podman[67531]: 2024-04-24 23:51:03.904518176 +0000 UTC m=+2.990828955 container died dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-24T23:51:04.547 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:04 smithi170 podman[67531]: 2024-04-24 23:51:04.271406922 +0000 UTC m=+3.357717711 container cleanup dfb83139b97fa4a9e211860d96d32213383322710ad06a3c83a609b1c73604d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T23:51:04.547 INFO:journalctl@ceph.mds.h.smithi170.stdout:Apr 24 23:51:04 smithi170 bash[67531]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-h 2024-04-24T23:51:04.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.h.service' 2024-04-24T23:51:04.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:04.733 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-24T23:51:04.733 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-24T23:51:04.734 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.k 2024-04-24T23:51:05.086 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:04 smithi170 systemd[1]: Stopping Ceph mds.k for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:05.086 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: 2024-04-24T23:51:05.066+0000 7f2caed24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:05.086 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:05 smithi170 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k[62106]: 2024-04-24T23:51:05.066+0000 7f2caed24700 -1 mds.k *** got signal Terminated *** 2024-04-24T23:51:08.158 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:07 smithi170 podman[67652]: 2024-04-24 23:51:07.89979635 +0000 UTC m=+2.938215924 container died bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T23:51:08.516 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:08 smithi170 podman[67652]: 2024-04-24 23:51:08.250234544 +0000 UTC m=+3.288654120 container cleanup bd204dd298897945fd13fe2c7fe5ebfdc175470566c754cb0aec2cd4486314da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-24T23:51:08.516 INFO:journalctl@ceph.mds.k.smithi170.stdout:Apr 24 23:51:08 smithi170 bash[67652]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-k 2024-04-24T23:51:08.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.k.service' 2024-04-24T23:51:08.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:08.701 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-24T23:51:08.701 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-24T23:51:08.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.c 2024-04-24T23:51:09.062 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:08 smithi195 systemd[1]: Stopping Ceph mds.c for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:09.317 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: 2024-04-24T23:51:09.087+0000 7fec58acd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:09.318 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:09 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c[63567]: 2024-04-24T23:51:09.087+0000 7fec58acd700 -1 mds.c *** got signal Terminated *** 2024-04-24T23:51:12.270 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:11 smithi195 podman[68187]: 2024-04-24 23:51:11.885467185 +0000 UTC m=+2.994101774 container died c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-24T23:51:12.562 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:12 smithi195 podman[68187]: 2024-04-24 23:51:12.379548951 +0000 UTC m=+3.488183541 container cleanup c6fb4eaf38e41a3afd6f562d6794e636f18a91de407202e46b0173d3f18915d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-24T23:51:12.562 INFO:journalctl@ceph.mds.c.smithi195.stdout:Apr 24 23:51:12 smithi195 bash[68187]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-c 2024-04-24T23:51:12.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.c.service' 2024-04-24T23:51:12.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:12.841 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-24T23:51:12.841 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-24T23:51:12.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.f 2024-04-24T23:51:13.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:12 smithi195 systemd[1]: Stopping Ceph mds.f for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:13.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: 2024-04-24T23:51:13.149+0000 7f55b63fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:13.312 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:13 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f[63262]: 2024-04-24T23:51:13.149+0000 7f55b63fe700 -1 mds.f *** got signal Terminated *** 2024-04-24T23:51:16.215 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:15 smithi195 podman[68319]: 2024-04-24 23:51:15.886212695 +0000 UTC m=+2.830925214 container died ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T23:51:16.215 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:16 smithi195 podman[68319]: 2024-04-24 23:51:16.216483674 +0000 UTC m=+3.161196189 container cleanup ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-24T23:51:16.561 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:16 smithi195 bash[68319]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f 2024-04-24T23:51:16.562 INFO:journalctl@ceph.mds.f.smithi195.stdout:Apr 24 23:51:16 smithi195 podman[68333]: 2024-04-24 23:51:16.434254147 +0000 UTC m=+0.541049348 container remove ff46aeac811e37078cbc9445a3b31f9e8d7d145b85e3f9bd4bd82bbb3f0e1aaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-24T23:51:16.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.f.service' 2024-04-24T23:51:16.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:16.726 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-24T23:51:16.726 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-24T23:51:16.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.i 2024-04-24T23:51:17.045 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:16 smithi195 systemd[1]: Stopping Ceph mds.i for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:17.311 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: 2024-04-24T23:51:17.044+0000 7f2f279f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:17.312 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:17 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i[63886]: 2024-04-24T23:51:17.044+0000 7f2f279f8700 -1 mds.i *** got signal Terminated *** 2024-04-24T23:51:20.062 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:19 smithi195 podman[68440]: 2024-04-24 23:51:19.891978153 +0000 UTC m=+2.951515948 container died ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T23:51:20.562 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:20 smithi195 podman[68440]: 2024-04-24 23:51:20.353295877 +0000 UTC m=+3.412833664 container cleanup ae6abb967957adab4faa6652b1a97aa5e71c95d1ec4e3fa2c85f78eeddb11a24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T23:51:20.562 INFO:journalctl@ceph.mds.i.smithi195.stdout:Apr 24 23:51:20 smithi195 bash[68440]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-i 2024-04-24T23:51:20.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.i.service' 2024-04-24T23:51:20.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:20.752 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-24T23:51:20.752 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-24T23:51:20.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.l 2024-04-24T23:51:21.061 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:20 smithi195 systemd[1]: Stopping Ceph mds.l for 16fa2ae8-0293-11ef-bc93-c7b262605968... 2024-04-24T23:51:21.561 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: 2024-04-24T23:51:21.064+0000 7fdd4d0ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T23:51:21.562 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:21 smithi195 ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l[62948]: 2024-04-24T23:51:21.064+0000 7fdd4d0ed700 -1 mds.l *** got signal Terminated *** 2024-04-24T23:51:24.190 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:23 smithi195 podman[68563]: 2024-04-24 23:51:23.891344505 +0000 UTC m=+2.927997901 container died b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-24T23:51:24.490 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:24 smithi195 podman[68563]: 2024-04-24 23:51:24.290468361 +0000 UTC m=+3.327121763 container cleanup b9c9e6ddeabaf9cdd25888207065fa7e9d921f8d6fa87de38056c2566e0ef507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T23:51:24.490 INFO:journalctl@ceph.mds.l.smithi195.stdout:Apr 24 23:51:24 smithi195 bash[68563]: ceph-16fa2ae8-0293-11ef-bc93-c7b262605968-mds-l 2024-04-24T23:51:24.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16fa2ae8-0293-11ef-bc93-c7b262605968@mds.l.service' 2024-04-24T23:51:24.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T23:51:24.668 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-24T23:51:24.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T23:51:24.857 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:27.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T23:51:27.427 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:29.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T23:51:30.074 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:32.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:51:32.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:51:32.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T23:51:32.465 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-24T23:51:32.699 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi107/crash 2024-04-24T23:51:33.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash -- . 2024-04-24T23:51:33.260 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T23:51:33.261 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-04-24T23:51:33.470 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi170/crash 2024-04-24T23:51:33.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash -- . 2024-04-24T23:51:33.740 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T23:51:33.741 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-04-24T23:51:34.052 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi195/crash 2024-04-24T23:51:34.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash -- . 2024-04-24T23:51:34.323 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T23:51:34.323 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-04-24T23:51:34.324 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-24T23:51:34.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-24T23:51:34.637 INFO:tasks.cephadm:Compressing logs... 2024-04-24T23:51:34.638 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:51:34.640 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:51:34.643 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:: No such file or directory 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-24T23:51:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.a.log 86.2% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 90.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-24T23:51:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log: 94.6% 83.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T23:51:34.671 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-24T23:51:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T23:51:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T23:51:34.682 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr: 86.1% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.2% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.9% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-24T23:51:34.683 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr: 86.1% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- 90.2% /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T23:51:34.684 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.3% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-24T23:51:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-24T23:51:34.688 INFO:teuthology.orchestra.run.smithi107.stderr: 91.1% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T23:51:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T23:51:34.690 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T23:51:34.691 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T23:51:34.693 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-24T23:51:34.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T23:51:34.702 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-24T23:51:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-24T23:51:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-24T23:51:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/cephadm.log: 94.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-24T23:51:34.713 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log: 92.6% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-24T23:51:34.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T23:51:34.716 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-24T23:51:34.717 INFO:teuthology.orchestra.run.smithi195.stderr: 92.2% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-24T23:51:34.717 INFO:teuthology.orchestra.run.smithi195.stderr: 91.1% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T23:51:34.717 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T23:51:34.717 INFO:teuthology.orchestra.run.smithi107.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T23:51:34.719 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T23:51:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-24T23:51:34.726 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-24T23:51:34.751 INFO:teuthology.orchestra.run.smithi170.stderr: 94.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-24T23:51:34.757 INFO:teuthology.orchestra.run.smithi195.stderr: 94.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-24T23:51:34.825 INFO:teuthology.orchestra.run.smithi107.stderr: 89.9% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-24T23:51:34.935 INFO:teuthology.orchestra.run.smithi170.stderr: 96.5% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-24T23:51:34.967 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-24T23:51:34.968 INFO:teuthology.orchestra.run.smithi195.stderr: 91.9% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-24T23:51:35.299 INFO:teuthology.orchestra.run.smithi107.stderr: 90.7% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-24T23:51:36.322 INFO:teuthology.orchestra.run.smithi170.stderr: 95.0% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-24T23:51:38.253 INFO:teuthology.orchestra.run.smithi195.stderr: 92.8% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-24T23:51:38.318 INFO:teuthology.orchestra.run.smithi195.stderr: 92.9% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-24T23:51:38.522 INFO:teuthology.orchestra.run.smithi170.stderr: 92.6% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-24T23:51:38.890 INFO:teuthology.orchestra.run.smithi107.stderr: 92.8% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-24T23:51:39.000 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-24T23:51:39.523 INFO:teuthology.orchestra.run.smithi107.stderr: 92.8% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-24T23:51:39.780 INFO:teuthology.orchestra.run.smithi107.stderr: 92.9% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-24T23:51:40.338 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-24T23:51:40.414 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-24T23:51:40.735 INFO:teuthology.orchestra.run.smithi107.stderr: 92.7% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-24T23:51:40.737 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-24T23:51:40.737 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m6.083s 2024-04-24T23:51:40.737 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m20.410s 2024-04-24T23:51:40.737 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.726s 2024-04-24T23:51:40.911 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-24T23:51:40.913 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-04-24T23:51:40.913 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m6.256s 2024-04-24T23:51:40.913 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m21.713s 2024-04-24T23:51:40.913 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.723s 2024-04-24T23:51:41.488 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/16fa2ae8-0293-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-24T23:51:41.491 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-04-24T23:51:41.491 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m6.831s 2024-04-24T23:51:41.491 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m19.447s 2024-04-24T23:51:41.491 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.725s 2024-04-24T23:51:41.491 INFO:tasks.cephadm:Archiving logs... 2024-04-24T23:51:41.491 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi107/log 2024-04-24T23:51:41.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:44.817 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi170/log 2024-04-24T23:51:44.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:48.142 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi195/log 2024-04-24T23:51:48.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:51.494 INFO:tasks.cephadm:Removing cluster... 2024-04-24T23:51:51.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force 2024-04-24T23:51:51.685 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:52.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force 2024-04-24T23:51:52.220 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:52.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16fa2ae8-0293-11ef-bc93-c7b262605968 --force 2024-04-24T23:51:52.690 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: 16fa2ae8-0293-11ef-bc93-c7b262605968 2024-04-24T23:51:52.966 INFO:tasks.cephadm:Removing cephadm ... 2024-04-24T23:51:52.966 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T23:51:52.984 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T23:51:53.001 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T23:51:53.017 INFO:tasks.cephadm:Teardown complete 2024-04-24T23:51:53.018 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T23:51:53.030 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T23:51:53.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:51:53.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:51:53.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:51:53.144 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T23:51:53.154 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T23:51:53.154 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:51:53.157 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:51:53.159 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:51:53.171 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:51:53.172 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:51:53.173 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:51:53.175 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:51:53.175 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-04-24T23:51:53.176 INFO:teuthology.orchestra.run.smithi170.stdout:^- hv01.front.sepia.ceph.com 2 6 377 29 +909us[ +909us] +/- 572ms 2024-04-24T23:51:53.176 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 +91us[ +88us] +/- 63ms 2024-04-24T23:51:53.176 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -108us[ -110us] +/- 65ms 2024-04-24T23:51:53.176 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:51:53.177 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:51:53.177 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-04-24T23:51:53.177 INFO:teuthology.orchestra.run.smithi107.stdout:^- hv01.front.sepia.ceph.com 2 6 377 30 +916us[ +908us] +/- 572ms 2024-04-24T23:51:53.177 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 +75us[ +67us] +/- 63ms 2024-04-24T23:51:53.177 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -123us[ -123us] +/- 66ms 2024-04-24T23:51:53.178 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:51:53.178 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:51:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-04-24T23:51:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +929us[ +925us] +/- 572ms 2024-04-24T23:51:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 +92us[ +88us] +/- 63ms 2024-04-24T23:51:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -101us[ -105us] +/- 65ms 2024-04-24T23:51:53.179 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:51:53.179 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T23:51:53.190 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T23:51:53.191 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T23:51:53.201 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:51:53.296 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:51:53.361 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:51:53.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:51:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:53.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:51:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:53.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:51:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:53.509 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T23:51:53.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T23:51:53.533 INFO:teuthology.task.internal:Duration was 1455.107053 seconds 2024-04-24T23:51:53.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T23:51:53.545 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T23:51:53.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:51:53.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:51:53.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:51:53.584 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:51:53.586 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:51:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:51:53.913 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T23:51:53.913 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-04-24T23:51:53.913 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:51:53.975 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-04-24T23:51:53.975 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:51:54.005 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-04-24T23:51:54.006 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:51:54.026 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T23:51:54.026 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:51:54.029 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:51:54.048 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:51:54.092 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T23:51:54.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:51:54.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:51:54.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:51:54.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T23:51:54.513 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T23:51:54.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:51:54.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:51:54.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:51:54.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T23:51:54.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.670 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-04-24T23:51:54.679 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-04-24T23:51:54.683 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-04-24T23:51:54.713 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:54.747 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:54.772 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:54.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T23:51:54.819 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T23:51:54.820 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi107 2024-04-24T23:51:54.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:51:54.896 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi170 2024-04-24T23:51:54.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:51:54.974 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672204/remote/smithi195 2024-04-24T23:51:54.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:51:55.023 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T23:51:55.024 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:51:55.027 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:51:55.030 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:51:55.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T23:51:55.108 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T23:51:55.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T23:51:55.137 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T23:51:55.138 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:51:55.140 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:51:55.143 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:51:55.153 INFO:teuthology.orchestra.run.smithi107.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:51 /home/ubuntu/cephtest 2024-04-24T23:51:55.155 INFO:teuthology.orchestra.run.smithi170.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:51 /home/ubuntu/cephtest 2024-04-24T23:51:55.157 INFO:teuthology.orchestra.run.smithi195.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:51 /home/ubuntu/cephtest 2024-04-24T23:51:55.159 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T23:51:55.191 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T23:51:55.232 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} duration: 1455.107052564621 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-24T23:51:55.232 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:51:55.311 INFO:teuthology.run:pass